Dave liked to think that he was a responsible, thorough application developer. He always tried to understand a problem before tackling it, to think through all ways a change could affect things, and to gather information before making decisions. So when he received complaints about the speed of the custom web application used by his work’s health department, he decided to do a little research to gather all the facts before reporting the issue to his boss.

Do not throw in trash

And sure enough, all it took was a little research: the logs were full of database query timeouts and memory limit terminations. After a small amount of performance analysis on the test database server, it was very clear that the database was by far the main cause of the slow speeds. As the number of entries in the database naturally increased, many of the queries became exponentially slower.

Satisfied he had actionable information, Dave went to his boss to get approval to make fixes.

“Thank you, Dave! But the database isn’t your job - you’re an application developer. Anything related to the database is the DBA’s job. I’ll get ‘Awesome’ on this right away.”

‘Awesome’, the affectionately-nicknamed-by-the-boss local Database Administrator (DBA), had no experience programming, but was considered “master of all things even tangentially database-related”. Dave tried to make his boss aware of an important caveat: “But the data access code is in the application. How can the DBA optimize it there?”

His boss was resolute: “That. Is. Not. Your. Job. Don’t even worry about it. I’m sure Awesome knows how to do it.”

And so instead of optimizing queries, Dave begrudgingly set up another testing instance of the application for Awesome to work with and handed over the details. Three days later, Awesome showed up at his door with bad news: “The application is completely broken.”

Incredulous, Dave logged in to debug a little and quickly discovered that the “web.config” file had been modified and was pointing to an incorrect server. “Awesome, please do not modify any code or application settings without letting me know. Modifying the web.config file can be dangerous if you don’t know what you’re doing. Just optimize by profiling the database on the testing database server - you don’t need to dig through the code.” With a glare, Awesome left, never to be heard from again.

Until 5 days later, when he showed up at Dave’s door with amazing news: “It’s done.” Again incredulous, Dave logged into the test instance… and was shocked by how fast it was. Every page loaded nearly instantly. Even complex records searches displayed results immediately. Not a single perceptible delay remained.

“How did you do it?”, Dave asked.

Awesome was recalcitrant: “You don’t need to know. It’s not your job. All you need to know is that it’s a lot faster, and I did my job.”

Dave pressed him further. Had he modified any code? Had he added any indexes? What changes had he made?
“No, no changes in code, like you told me. The code is your job”, Awesome calmly informed Dave. Honestly puzzled, Dave pressed one more time for details: “Well… what exactly did you do?”

“Oh, the old database server was really slow, so I copied the database to a new server with a lot more RAM.”

“So copying the database to another server took 5 days?”, Dave asked.

“Well, I also had to archive some old records. The database had too much data in it.”

And that’s when Dave discovered the real reason things were so fast on the test instance: Awesome had “optimized” the database by simply removing a lot of data from it. Were it full of logs, or other data that had a defined shelf-life, this might have been acceptable. But as the database was full of the health department’s records that had to be maintained, this was pretty much the definition of Not Okay™.

“Awesome, we can’t just delete data from the database. We have to maintain those health records.”

At least the changes hadn’t been deployed to production.

“… But… I already made the switchover to the new database server.”

And that’s when Dave checked the production instance and realized that it, too, was similarly fast. And that it, too, was similarly missing lots of records.

As luck would have it though, Awesome was not very thorough at his job, and he hadn’t bothered to touch the old database on the original server. So Dave logged in, changed the configuration to point at the original location, and the site came back up with all old records intact… and without the “optimizations” that Awesome had put in place. He informed Awesome: “No worries - I’ve fixed it. All the data is back. Just leave the new server up for a bit so all the changes can sync. I’ll let you know when it’s done.”

As Awesome walked away, confused but relieved, Dave settled in for a long day’s work. Someone was going to have to manually merge any new entries and records changes back into the original database - and then actually improve performance. And, while it wasn’t his job, Dave figured that those tasks might be better left to someone a bit more understanding and thorough, instead of someone “awesome”.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!