Peter was cautiously optimistic before the busiest part of open class enrollment season at his employer - Northeastern Institute of WTF. Peter and his team worked hard to implement a new user-friendly website for students to enroll in classes. Now, the brain-dead students who couldn’t figure out how to register should call the service-desk less. He even contracted a third-party datacenter with “state-of-the-art load balancing hardware” to handle the web traffic stress on the system. This would be the simplest enrollment season ever!

Three days before the enrollment deadline, all the procrastinating students (i.e. all the students) swarmed the website at once. Instead of being greeted with the shiny new enrollment site, they found the dreaded “Page Unavailable” screen - if they even got any response at all. This in turn loaded the IT support office’s phone lines with calls of “OMG I waited until the last minute to register for classes and now I can’t!!! I NEED Survey of Underwater Basket-Weaving or I won’t graduate!”

Peter and his staff of three got overwhelmed, so they flipped the switch on the phone system, activating the “We’re sorry, all lines are busy right now” prompt to stop the calls. With this chance to breathe and figure out the problem, Peter dug in to the network logs. There where myriad connection errors. The student traffic got to the remote load balancer but didn’t get a response.

Peter got on the horn with the datacenter support staff and explained the problem. A man named Roger gave him the runaround from the datacenter, denying it was a problem on their end. Through Peter’s persistence, he finally got them to agree to look at the load balancing device and servers it routed to. “Now we play the waiting game…” Peter said to his guys as they started up a foosball match while the department’s phones were still ignoring everyone’s calls.

After what could have been an entire World Cup’s worth of foosball, Roger finally called back. “I’ve got bad news and bad news. Which would you like to hear first?” Roger snarkily said. “Um… bad news?” Peter said before gulping.

“Our load balancer is using the default settings, which is simple round-robin,” Roger explained. “Requests alternate between the two servers, regardless of how loaded those servers are.”

“Ok, so you can just change that setting and everything will be ok, right?” Peter replied with hope.

“Well now, that’s the other bit of bad news. We also have a randomized server backup system in which one of the servers in our datacenter is chosen each week to handle backing up all the other servers. Unfortunately, the server this week is one of yours. We aren’t able to stop the backup without messing things up, and it has 18 terabytes more to get through. So, it’s going to be about 48 hours before your server farm can handle anything other than mild stress,” Roger stated ominously.

When Peter hung up, he felt like he’d been gut-punched. He instructed his subordinates to draft an e-mail to all students and faculty to explain the open enrollment system wouldn’t be usable before the end of the week. Peter took on the task of crafting an e-mail to the dean of Northeastern Institute of WTF with about 25 apologies and a request that the class enrollment period be extended a few days. Peter then put his head down on his desk and considered firing himself for not vetting the third-party datacenter before signing the contract. This suddenly became the most difficult enrollment season ever.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!