In the early part of this century, it was quite common for small companies to stray into the infrastructure business and run their own servers. The company where Tatiana's dad worked was no exception. The infrastructure in this case was mainly a couple of old Linux boxes, one acting as the Web server and the other as the email server. To be fair, the email server was 'newer', in that an older one had recently been replaced. Imagine for a moment what the specs for an 'old' server in 2000 that was too slow to support an email server must have looked like. I imagine a 186 chip and 8K of RAM with a cassette tape for persistence, but that could just be me.

At the time, Tatiana was a university student with a working knowledge of Unix. Where, in this case, 'working' is defined as the level of UNIX learned while completing university-level computer courses. Still, when the system administrator at her dad's company left and her dad asked if she would be willing to help out if an emergency arose, Tatiana agreed.

One evening she received a call from her father saying that the server was acting up. "But Dad, I'm out with my friends now..." she said.

"It's fine, take your time, but don't forget to look at it as soon as you're home tonight!"

When she got home, she logged into the server and relatively quickly found that the /var partition was filled to capacity. While her first thought was old logs, it turns out that the black hole of disk space was the mail queue. A quick inspection revealed that the vast majority of those emails were trying to be sent to the previous sysadmin's email address and the server used to host that email seems to have been decommissioned. Tatiana received permission to flush the whole mail queue and, upon completion, she went to sleep, hoping to be awake enough for next morning's lectures.

The loss of a few emails was annoying, naturally, but not the end of the world. Still Tatiana had a bad feeling about the server. What was the source of the maelstrom of emails? Suggestions for a better collective noun can be made in the comments. Her nature did not allow her to just drop the issue. She kept monitoring the mail queue and, as she anticipated, it started to grow, slowly but surely. Rather than wait for the inevitable phone call, Tatiana decided to prophylactically tackle the issue. While the directory wasn't going to fill up in minutes, the era in which the hard drive was forged was such that it would be weeks and not months before the problem reappeared.

She soon found out that the previous sysadmin's mails were being forwarded to the non-working mail server. What's worse, root and postmaster were aliased to the previous sysadmin's user name. She fixed that, cleaned up a few broken cron tasks, and monitored the mail queue until she made sure that it was able to clean itself up.

Tatiana took a deep breath and started thinking about how the system was structured. Everything that went wrong in the system, including the broken cron tasks, would send an email message to root. And the sender of the message was also root. The mail server would try to deliver the emails for a number of hours. When that didn't succeed. two warning emails would be sent: one to postmaster and one to the sender (root). The original emails would then sit in the mail queue for a couple of days. Just in case the server came back on-line. Once two days had passed, two failure emails would be sent. One to postmaster and one to root. Which would, naturally, start the process over again. But with four times the number of warning and failure emails as before.

After Tatiana's dad heard the explanation of what had happened, he asked for Tatiana's account and root access to not be revoked in the future, even after they found a new sysadmin. As a bonus for her, she'd have a small web space and email address for personal use. 13 years later, the mail server has been deployed onto new hardware, but Tatiana's account and root access still exist. When asked for a suggestion for the name of the new hardware, Tatiana suggested "Faberge"

 

photo credit: sixelsid cc

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!