Anna’s employer made billing software for energy companies, and they had just entered into an exciting time: the Dutch utility market was loosening its regulations, and that meant a flood of startups were looking for a cheap way to bill their customers. Of course, their product was designed for the US market, not the Dutch market, but the CEO didn’t see that as a problem. “No worries,” he would say, “Anna’s our top notch configurator, and she’ll customize it for you in no time.”

Electrical Hazard (wiring)

This, of course, was news to Anna, who didn’t actually know anything about the configuration of the software. After a one-day training session, she was officially declared the “expert”, and sent off to sink or swim- and she mostly swam. There were a lot of options in the package, and it was extensible through add-ons, which meant Anna spent a lot of time whipping together one-off modules for each customer’s deployment.

During this period, one of her “favorite” customers was Joep. Joep had needs. Joep had a lot of needs. He was polite, and friendly, and Anna liked Joep, but Joep tended to start every conversation with, “You know, it’d be nice if…” and then invent a thousand new requirements for the billing package.

A lot of them were around reporting, which was not the system’s strong suit. Joep wanted the ability to run ad hoc reports against any dataset his mind could imagine. “How many left-handed customers logged into the billing application between 2:04AM and 2:37AM and were also watching something on Youtube?”

This lead to a lot of screen-sharing sessions and on one of those calls, Anna fired up Toad to connect to the back end database.

“Oh! What’s that?” Joep asked, as she typed in a query.

“I’m just trying to see if we’ve got the data we’d need to build your report.”

“And you can type commands… to build custom queries?”

“Yes…” And Anna was seduced by temptation. Joep wanted a lot of custom reports, and if she showed him how to use Toad and introduced him to basic SELECT statements, he’d be able to “self service”, and that would save her a lot of headaches. So Anna helped him install Toad, disable the auto-commit functionality, and showed him how to write SELECT statements. Joep was happy, and Anna fielded far fewer calls from him.

Time passed. Anna moved on to work with other customers, Joep changed jobs, and roughly a year later, their paths crossed again. Anna’s latest client was Joep’s new employer. They shared a laugh, reminisced about old times, and then talked about Joep’s new needs.

This time, they had a SQL Server back-end, which didn’t really change much about how the billing package worked. Anna noted that, and started gathering requirements for the new batch of add-ons and custom screens she’d need to build. While she was concentrating on that, Joep asked, “I was looking at SQL Server, and Microsoft’s management tools look even nicer than Toad. Do you think I could use those?”

“Sure, I guess,” Anna said, without really thinking about it. Over the next few weeks, she got Joep’s new company set up and configured to their liking, and moved on to the next client.

Months later, this email arrived:

From: [email protected]
Subject: FIIIRRREEEE!!!!!!!!!!!!!
CALL. ME. NOW.

Anna called. Joep picked the phone up before Anna even heard it the ringtone, and screamed, “WHERE IS THE ROLLBACK? HOW DO I ROLLBACK? ARGHTNHR!”

Anna, of course, had only taught Joep about the SELECT statement, but Joep was a smart guy. He went on to learn all about commands like UPDATE, DELETE and TRUNCATE. Joep liked to run “hypothetical” reports from Toad, by altering the data. When she had him setup Toad, she was very clear on how to prevent Toad from auto-committing changes.

The SQL Server Management Studio, and SQL Server in general, defaults to automatically committing each statement.

“I was just trying to get rid of the test accounts,” Joep sobbed.

Anna got Joep to share his screen, and tried to understand what had happened. Joep had gotten in the habit of running commands in SSMS by selecting the statement he wanted to run and hitting F5, and this time he got a little sloppy with the mouse- instead of selecting every line in: 

DELETE FROM customers
WHERE account_name LIKE "TEST_%"

He only selected the first line. 12,000 customers were wiped out in an instant. There was little Anna could do- she sent him off to talk to their DBAs. Even after restoring from backup and reviewing the logs from the web application, they lost somewhere between 1,000 and 2,000 customers.

And so, Joep moved on to his next job.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!