Andrei had just moved to a new country, and was looking for work. With the ideal job not immediately forthcoming, he jumped when an interesting opportunity opened at a small tech company: OldIsNewCo.

OldIsNewCo was one of the pioneers bridging the gap between old methods of communication (e.g. paper) and new methods of communication (i.e. the Internet), and was looking to overhaul their entire infrastructure. Their existing backend was built in C++, and according to The Big Boss: “C++ developers are expensive and hard to find. PHP developers are plentiful and cheap. Therefore, we need to re-implement everything in PHP.”

This seemed fishy to Andrei right from the start, but he figured it would be a good opportunity to brush up on his C++ and PHP skills and experience, and “a job’s a job”. So he accepted the position and met with his development tour guide to start diving into the sources and seeing how the backend applications worked.

Or, he tried - only to quickly discover Problem #1: There wasn’t a complete version of the source code in any single location. The sources for the backend were spread out over different production machines (i.e. the machines running the compiled code). Because these machines had variations, building was done on each machine locally, since the code might not build on a different machine with different software or configuration. This contributed to Problem #2: Making modifications to some of the backend was done by editing code on the same machine where that component ran.

As Andrei wrapped his head around those concerns, he began to wonder how the developers tested their changes. His development tour guide responded: “Oh, we don’t have a testing environment. We just deploy the new version to the live servers.” Deploying to production consisted of: waiting until the night when the system was mostly free of clients, compiling the code on the same machine it would then run, and hoping for the best. If the new build crashed the next morning when users returned, they would just revert to the old build and repeat the attempt the next evening.

But project organization, testing and deployment were far from the only concerns. As he finally began to dig into the code and architecture itself, Andrei discovered that the existing C++ backend had a few problems of its own:

  • The core backend app simply operated on files - it read them in, processed the contents, and output data to a database and other machines via the network. But this was implemented in the most complicated way possible, with numerous multithreading issues that resulted in the deployed application crashing frequently
  • It crashed so frequently, in fact, that it was one employee’s main job to watch the servers, wait for the app to crash, and manually restart it.

  • All database synchronization was done in the application’s own C++ code, instead of using the built-in synchronization tools that were part of the database engine (i.e. no database transactions or locks were used). Combined with poor database design, this part of the system would also overload and deadlock quickly. As such, it was another employee’s main job to monitor for this, kill all the processes, clean up the databases by hand (because they’d likely have incomplete or corrupt data), and manually restart everything.

  • Access to the database itself had enough pieces hard-coded in the C++ code that each time anything database-related was modified the C++ code had to be searched and modified extensively too. And if everything that a change affected wasn’t updated, everything would crash the next day when the untested code was deployed to production.

  • The database was supposed to be replicated, but actually was not. This was discovered when the main database server crashed and the replicated database responded with errors and missing data.

  • To meet the “new requirements” of The Big Boss, many of the latest features were allegedly implemented in PHP code but were actually coded as a series of thousands of system calls from PHP. (Apparently no one knew how to perform file management or monitoring tasks from either C++ or PHP code.)

  • The existing code-base was written using ancient libraries running on an already-set-to-be-imminently-discontinued version of a Linux distribution. So the servers couldn’t be upgraded without extensive rewrites anyway.

Shocked by the state of everything, Andrei decided to kick off his rewrite project on the right track, and work up a proposal to take a reasonable number of days to collect and centralize the sources, create development / test / deployment environments and scripts, and otherwise organize the existing project. But his development tour guide quickly took him aside:

“I made the same exact proposal, and it wasn’t just denied - it’s the reason I’m being let go in two weeks. According to the boss, I proposed: ‘doing nothing for 2 weeks’.”

It was then that Andrei learned the underlying reason for a lot of the problems: The Big Boss deemed useful only those developers who could meet his demands immediately. Any other proposed steps or actions were seen as potentially fireable “wastes of time”. Fellow employees were so afraid of The Big Boss that they gave up on all “time wasting” tasks, including organizational upkeep, sensible deployment or testing practices, or even sometimes coding things the proper way (because the deadline was whatever The Big Boss deemed doable).

So Andrei sighed and kept his head down. Piece by piece, he ripped out chunks of the old C++ code and replaced them with lightweight modules written in PHP that were considerably faster, simpler, smaller, and actually stable. When he had extra time within the deadlines assigned to him, he even set up his own private development, testing, and source code versioning environment to make his task easier (but he couldn’t tell anyone else - let alone The Big Boss). Meanwhile, every 2–3 weeks a developer or new-hire who couldn’t implement the boss’s whims quickly enough was fired, especially those who dared to propose spending time on testing or documentation or other “wastes of time”.

When Andrei finished the rewrite a month early, the backend system’s codebase was several times smaller, ran several times faster, and required 1/4 of the servers to run without crashing. The Big Boss was pleased and asked him to “take a quick look at the databases - you know, tune them up, make them run a little faster”, but Andrei had already found another job, and quit OldIsNewCo - eager to leave the memories of their Horrible Software in his past. After all, with a new job on the line, staying would have just been a “waste of time”.

Image source: 1, 2

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!