It was the calm before the storm. Brokers were sitting at their desks in silence, watching the clock. The market was going to open in minutes, and huge volume orders would start pouring in. The developers working for the firm – a mid-size proprietary trading outfit on Wall Street – were already busy; an order from the previous day should’ve expired automatically, but didn’t. It was manually fixed moments after it was discovered.

“Huh,” Daniil shrugged, “I wonder if this has anything to do with the latest release.” They’d just rolled out a minor update to their proprietary trading system. Daniil had overheard his boss barking at a junior developer that they needed the feature in two days – testing, review, standard processes be damned. “Done” had a higher priority than “working.” Any objections raised by the developers were met with the same reaction you’d get from a dog after explaining the Pythagorean Theorem – blank, drooling faces.

DING DING DING DING – the market was open. All eyes turned to the board that’d soon populate with dozens of orders. Brokers glanced at the clock and then quickly back to the board. Any second now the avalanche would begin. Any second now…

Nothing happened. Brokers’ heads started poking up above cubicle walls like gophers, each one looking more panicked and confused than the last. The panic was contagious – developers were in a panic trying to figure out why there weren’t any orders. Developers clicked, typed, and swore like crazy to locate the source of the issue, and it only took a few minutes. It was a bug from the latest release.

They couldn’t roll back since the market had already opened. The only option was to fast-track some changes, putting the system at even more risk. Brokers egged them on – “there’s no time to test! Just get it working,” they urged.

The attitude among the devs was a strange sort of frenzied optimism. “We might even have this fixed before any clients even notic-“ The phone started ringing. It was the CEO.

“What the hell is happening up there? Our orders are getting rejected. Why?”

Daniil did all he could to reassure him that it was being addressed, cut short by the CEO being screamed at in the background, then yelling a hurried “I’ll call you back.” Daniil feared that a bunch of angry brokers had gone vigilante and killed the CEO.

The panicked optimism gave way to plain old panic. Daniil looked up orders caught by the web service but not processed – in the span of two minutes, seven orders had been placed, each around $15,000. In 120 seconds they’d lost over $100,000 of the orders. Other developers shouted that they were discovering errors on other systems.

What’s worse, sell orders were failing as well. One dev, using language that would make a sailor cry, told Daniil that he’d discovered a failed sell order that was now worth $5,000 less than it would have been just a few minutes ago.

Meanwhile, another dev had found and fixed the problem – a stored procedure had changed, and error conditions were never checked. The updated procedure was called by about a dozen other procedures in the system, and since there was no error checking, it’d cancel the call and continue right along as though nothing had happened. The error message would be displayed only after the order was already recorded, but not completed.

After running some numbers, Daniil discovered that the amount of lost commissions, sales, and trades outweighed the cost of testing and fixing the change over ten-fold. Mentioning this to his boss, it was promptly ignored. “Is the fix in production yet? What are you waiting for?! Deploy! DEPLOY!”