Most folks know that I've spent the last couple of years on Team-WTF in Department-WTF at WTF-Inc. A while back I moved on to BigCo Inc. This place is a huge bureaucracy, but somehow, things manage to get done. As with most places, it has its share of (sometimes epic) WTFs, but those can wait for another day. This time, I got to witness WTF-Inc - from The Other Side.

Due to government regulations, BigCo found itself in need of the sort of services provided by WTF-Inc. Since there are only two companies that provide this type of functionality, and the other company has a long way to go to catch up to WTF-Inc, BigCo didn't have many alternatives, so they decided to check out WTF-Inc. As part of it, someone at BigCo got the idea to search its huge HR database of resumes to see if one of its myriad employees has ever had any association with WTF-Inc. One name popped up.

They invited me to a meeting to pick my brains. And pick they did. Since I was somewhat leery of providing too much specific information on just how screwed up WTF-Inc is, I was a bit hesitant to offer many details, speaking instead in generalities. They insisted on particulars, so I told them why I was hesitant, and instead offered a list of very specific questions to ask, instructing them to look for verifiable answers. I told them to insist on specific demos, run by BigCo (not WTF-Inc). I told them to run tests that scale up to the expected level of throughput and verify that it works as expected and finishes within a reasonable SLA before committing any real money or resources to it.

The folks representing BigCo set up the demo with the sales drones from WTF-Inc. The sales drones basically yes'd them to death on all the questions, leading the BigCo folks to be suspicious. Then came the demo. First one record. Then 10. Then 100. Then 1,000. Then 10,000. Everything seemed to work. An initial contract and NDA to perform load testing were signed.

Since this was the full-scale test, BigCo sent 40,000,000 records. *cough* *choke* *sputter* The applications at WTF-Inc came crashing down as the database and JVMs ran out of space. Whoops!

The folks orchestrating the demo for BigCo called me; I told them what likely was happening, and to demand that the sales-folks get it resolved before moving forward. Over the next several days, I got wind of emails from the sales drones to BigCo explaining that they've made some performance enhancements to the application (my spies back at WTF-Inc know where I'm working; they called, and hysterically informed me that all of my code that had been reverted back to the way I wrote it) and that it will now handle the data load. Unfortunately, they didn't make the changes to the database that were required to handle this huge influx of data. *cough* *choke* *sputter*

The sales drones insisted it was no longer an application failure. BigCo said they didn't care where it failed, or why; only that it failed at all. About two weeks later, the drones called again, insisting that the database can now handle the large influx of data. Another test was set up. This time, the test ran to completion, but took 8 times the promised regulatory-specified SLA to finish. The drones insisted all was well. The BigCo folks called me again, asking if the SLA was unreasonably low. I told them that if it was taking 8 times as long in a dedicated demo environment, they should ponder how long it would take in an environment shared by all of WTF-Inc's other customers. Since BigCo was prohibited by regulatory edict from taking that long to provide these reports, they had to insist that WTF-Inc resolve their issues. In parallel, they told the regulatory authorities that the only available provider couldn't meet the regulatory requirements, and that while they were attempting to comply with the rules, BigCo was dependent upon WTF-Inc's ability to get their stuff in order.

I figure it will take two years for WTF-Inc to implement the necessary upgrades to their infrastructure to be able to handle this much data at once. By then, if their sole competitor has any brains, they will have swooped in and stolen all WTF-Inc's customers.

It's amazing to watch it happen from the other side...

 

Photo credit: tinyfroglet / Foter / CC BY

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!