Photo credit: 'digicla' at Flickr At my daytime corporate-type job, if I need to even sneeze in the general direction of a production environment, I need both a managerial and customer approvals with documentation solemnly stating that I thoroughly tested my changes and swear on a stack of MSDN licenses and O'Reilly books that I am NOT going to break anything as a result of my changes.  Sure, the whole thing is a pain (and admittedly, a necessary evil), but what Bruce W. has to go through beats the pants off of anything I've ever had to go through.

For the most part, Bruce loves his job.  He gets to work with a lot of intelligent and motivated people.  He has been developing a new system to support a new product that has the possibility of earning his division several million dollars per year and saving the corporate parent several hundred thousand dollars per year.  The net effect on the corporate parent's bottom line will be quite nice.  He developed a Web front end while a fellow developer put together the data feeds.  The initial development work was estimated to take about six weeks; pretty good since we only had eight weeks to work with. 

However, Bruce works in a very large corporation (70,000 plus employees through out the US and several countries) and IT for the corporation has been highly centralized to the world headquarters.  Smaller IT work, like the development and support for only a single division, isn't centralized but must pass through the central Mega Bureaucracy for approval and placement on the centralized servers.

...and Bruce needs their "help" to officially set up his environments.

You see, while Bruce and his group can test all day long on their local computers and servers, any kind of "live" environments must be created, blessed, and centralized by the Mega Bureaucracy.  They're bigger, badder, and have more connections than anybody in your division's rank-and-file.  Remember: in the Mega Bureaucracy, processes and procedures are to be followed, respected, and if necessary worshipped.  Oh, and forget even thinking of installing Web services on one of the existing centralized servers.  That would bring down the wrath of the entire blessed Bureaucracy for changing the purpose of an existing machine without first going through Mega Change Server Process.

Here's a brief overview of what Bruce had to go through to get four (one each for development, testing, staging, and production) Windows-based Web servers:

Week 1 - At the same time Bruce's group started the project he went to procure the servers.  He was told that all he needed to do was put in a Service Request with the Windows Server Team and they would get what we needed.  However, that request is cancelled because when the Windows Server Team saw that the servers were for a new application they said, "Whoa, you have violated rule #38,991 of the Mega Bureaucracy!  New applications must go through the Process for Application Implementation and Navigation." 
 

Bruce starts into the fill the first two PAIN forms (one being 20 pages long with 150 questions), sends them off to the server team, and immediately receives a response that, no, do not directly send PAIN forms to the group they go to.  Instead, open a project with the Mega Bureaucracy's project tracking system, attach the forms and THEN assign the project to the group.

A few days later, he receives word that the project has been accepted, slotted, and a project manager assigned.  Bruce figures, "Cool, now we are moving!  I'll have my servers in no time!"  He and his boss have a conference call with the PM and express to him the time critical nature of these servers.  The PM agrees to push them forward saying that the request isn't complex and shouldn't take much effort.


Week 2 - Bruce receives the initial project estimate and immediately replies with his approval.


Week 4 - Bruce calls the PM to find out what's going on.  He says that due to staffing cuts only a handful of requests are being processed at a time.  Despite being reminded that this project is literally worth millions, he says that other projects are ahead of us and that this is simply how things are.  Bruce boss escalates the issue to the head of IT for the entire division who just happens to be a member of the Project Approving Council and supposedly has the power to move the project forward.


Week 6 - Only three weeks until the promised delivery date, Bruce learns that the project still has not moved.  His boss fires off a series of emails saying that the app is about to go live on a system that will earn the company millions of dollars that is running on a desktop machine sitting in a cubicle.


Week 7 - The system is now fully coded.  Bruce is walking around, shaking his head, saying to himself "We have done user testing and end-to-end testing on a desktop machine-based server!"


Week 8 - The new system goes live and is serving dozens of customers daily.  The difference between Production and Test environments is a Post-it Note.  Power strips and network hub are carefully labeled "DO NOT TOUCH! HIGH VOLTAGE!" to prevent cleaning staff misfeance.


Week 10 - Bruce and the Windows Server Team finally have the project kick off meeting for the servers.  About 15 of the 30 minute call was spent with Bruce repeatedly saying, "All I need is a Windows Server with IIS and .NET.  I do not need a database server, no access to the mainframe, no massive SAN space, no Internet access, no interplanetary probe, just servers."  "BUT", they say, "You stated on page 16, question 113 that your application uses a database.  Where will that database come from?"  Bruce explains again, "We are using existing databases assigned to our group.  The database is outside of the scope of the project of setting up four Web servers."

Week 12 - Bruce and the Windows Server Team get together for their status meeting.  The server team says they haven't budged since last meeting.  Why?  Everyone says, "Well, we're just waiting for the other shoe to drop and this becoming a big, complex, hairy project requiring massive time."  Bruce once again states that all they he needs is four Web servers.  Nothing more. The server design engineer says, "Wow, that is pretty simple.  Shouldn't take too long at all."


Week 14 - Bruce has another status meeting with the PM and the server engineer.  The engineer has put together the required diagram of the requested infrastucture and states that he only had to change a handful of things from the initial template.  He says that everything should be ok and once they have the infrastructure readiness, the server builds can start.  Bruce thinks, "Finally!  All the other people initially assigned to the project must have realized that building four web servers isn't that big if a deal! ...haven't they?"


Week 18 - The head of IT for our division finds out that we are still waiting.  Heads start rolling...even poor Bruce's.  "WHY DIDN'T YOU CALL ME SIX %($$!*& WEEKS AGO???" the IT head blasts.


Week 19 - The servers are built (it only took 2 days to build them!) and are signed off for production support.

Week 20 - Bruce distributes the application URL pointing to the brand new servers.

Through all of this Bruce learned a couple things.  First, don't even think of going around the Mega Bureaucracy, even if somebody says you can.  The Mega Bureaucracy remembers and brands you a heretic.  Second, if you think you will need help from the Mega Bureaucracy, start early, fill out all of the forms, stand in the right lines, sacrifice to the appropriate gods, and don't even hint that you would think of going around them. Finally, he who yells loudest gets move the front of the queue soonest - as holy and almighty as The Mega Bureaucracy is, they're happiest to get rid of their crabbiest customers first.

The silver lining in all of this?  Apparently, the Guardians of the Mega Bureaucracy seem to now be willing to consider that there is a different tier of requests that don't require so many stopping points, designed to make sure that users really, REALLY know what they want to request.  Bruce remains positive saying that, maybe in a few years, after meetings to plan meetings, forms to request forms, they will have a process that only has an initial questionnaire of 10 pages and 75 questions.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!