83.3% Uptime

It was the klaxon noise that Jacky hated the most. If there was even a whisper of the word “hacked,” Mr. Cullen would sound the alarm.

“All-hands meeting, boardroom, right now,” Mr. Cullen said. He left the klaxon on as the employees of Red Feather assembled in the boardroom. “The webserver,” he said gravely, “is no longer responsive. We’re being hacked as we speak.”

Jacky suppressed a laugh.

“Jacky,” Mr. Cullen said, “you’re the sysadmin. You should be more concerned.”

“You’re absolutely right,” Jacky said, dropping his smile. “I’ll look into it.” Everyone else looked relieved as they left the boardroom; Jacky knew that it was merely DEFCON 2; if he failed to find the cause of the “hack,” things would go to DEFCON 1 in a hurry.

And the klaxon would remain on until the crisis was resolved.

Creaking Cron Tasks

Jacky logged into the webserver. Executing simple posix commands took thirty seconds apiece; whatever was hitting the server was hitting it hard. Jacky navigated to the error logs and got a dump.

[root@localhost ~]# grep '\[notice\]' /opt/apache/logs/error_log | less [Tue Mar 29 06:42:01 2011] [notice] caught 
SIGTERM, shutting down [Tue Mar 29 06:42:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming 
normal operations [Tue Mar 29 06:43:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:43:11 2011] 
[notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:44:01 2011] [notice] 
caught SIGTERM, shutting down [Tue Mar 29 06:44:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- 
resuming normal operations [Tue Mar 29 06:45:02 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:45:12 
2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:46:01 2011] 
[notice] caught SIGTERM, shutting down [Tue Mar 29 06:46:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured 
-- resuming normal operations [Tue Mar 29 06:47:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 
06:47:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 06:48:01 
2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:48:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 
configured -- resuming normal operations [Tue Mar 29 06:49:01 2011] [notice] caught SIGTERM, shutting down [Tue 
Mar 29 06:49:11 2011] [notice] Apache/2.2.14 (Unix) PHP/5.3.3 configured -- resuming normal operations [Tue Mar 29 
06:50:01 2011] [notice] caught SIGTERM, shutting down [Tue Mar 29 06:50:11 2011] [notice] Apache/2.2.14 (Unix) 
PHP/5.3.3 configured -- resuming normal operations ...........

Why is it shutting down so often? Jacky wondered. Is there an Apache exploit some black hat just found? Jacky didn’t think so. The server was going down for about ten seconds, coming back up, then faulting again fifty seconds later. It was so regular...like a scheduled task.

He dug into the cron scripts.

Periodic Downtime

The webserver had eleven kinds of cron tasks scheduled, in increasing frequency from every minute to 24 hours. Each cron script was fairly short, just a few lines each. He dumped the contents of the one-minute script.

*/1 * * * * cd /opt/apache/bin; ./apacheRestart >/dev/null 2>&1

Wait...What’s apacheRestart? Could it be...?

[root@localhost ~]# cat /opt/apache/bin/apacheRestart /opt/apache/bin/apachectl stop sleep 10 
/opt/apache/bin/apachectl start

Jacky couldn’t help but laugh.

Tolling Bells

The server was back a few minutes later, after Jacky commented out the line executing apacheRestart. He went to see Mr. Cullen.

“I think the firmware was updated recently,” Jacky said. “The server had been restarting every minute since the task was added. We just didn’t notice any downtime until Apache couldn’t stop properly.”

“Do you think,” Mr. Cullen said, “that a hacker could have--”

“--I doubt it,” Jacky said.

Mr. Cullen relaxed. He flipped a switch on his desk, shutting off the klaxon. “Thank you, Jacky.”

Jacky walked back to his desk and put on his headphones. He tried to play some music, but his ears were still ringing.

Photo credit: jeff_golden / Foter / CC BY-SA

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!