Originally posted by "snoofle" ...

I've mentioned a few times that I joined a team well into a two year development project. The system was designed, based on the sponsoring users claims, to handle 500-600 million records per day.

Today was deployment-day. Did we take time to set up a production environment? Well, they bought the servers and installed the OS. Cron-jobs? Who needs them. Did we take the time to test the DB's to make sure they were configured correctly? Nah, not necessary. Did we freeze a month ago, with only code changes resulting from testing? No. Was testing done? No. Was there even a test-plan? No.

The sponsoring users kept demanding logic changes as they figured out how the system was supposed to behave. Right up to 4PM this afternoon (5PM was go-live). Hell, there was even time today to add new connectivity to a new external data feed. It's not like we need to test those sorts of things or anything.

We wound up with a non-functioning ill-configured production environment where we simply couldn't fire up the application because the DB indices were AFU, and it was processing at the mind boggling speed of 1 record per second (apparently replicating the configs from the dev DB was not a viable option, don't ask me why). As such, the decision was made by our users to run everything from the dev DB (which was fairly zippy). The prod data would be copied to dev, and we'd just run the processes manually. Um, wait a minute. 500-600MM records into the very modest dev DB? Trash all the scratch tables. Trash all the test DB's. Hell, trash everything. It's not like developers need a place to work or test their code. Make all the room you can for that data.

Ok. 5PM rolls around. We get the go-ahead to copy the prod data to dev. ~350 records. Let me restate that: about-three-hundred-and-fifty records. 350-f'ing K of data. It took less than one second to process the whole thing. The output report was 1K. The whole thing would have fit on a motherf-----g floppy!

We send the output report to the sponsor who proudly sends it up the chain. The overlord responsible for the budget looks at it and exclaims "Two years and $MM's for THIS? WTF!?" He then orders all of us into a room to explain ourselves. Why we bought zillions in servers, TB's of san storage, etc. We trotted out the e-mails from the sponsors with the data estimates. They asked us why the hell we were running a prod job in dev? (BTW, our firm policy actually considers this a major no-no). We trotted out all the memos from the sponsors demanding changes right up until the last minute (also a major no-no). They asked us how we managed to re-test all of the components of the system in such a short time. "We didn't because we weren't given ANY time to do testing/stabilization" (mucho major no-no). Finally, they asked us how we even knew the damn thing met the specs. We said "what specs?" The overlords then turned to the sponsor and inquired as to how they could spend 50 man years on a project over two years and not have specs. The sponsors responded that we don't need specifications because it will be done when we don't see any more problems.

We were thanked for our time and told to go home. The sponsors stayed behind. Personally, I hope they were eaten.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!