The custom-built PHP-based content management system suffered from the classic problem of too many cooks in the kitchen. Every code file had conflicting naming conventions and coding styles, structures and duplicate methods all over the place; a Big Ball of Mud. And Dan S. was thrown head first into it.

Dan was trudging through the usual drudgery of low-to-medium priority tickets, when suddenly the heavens cracked open and a Very High priority ticket descended from the sky; the first he'd ever seen. A Very High priority ticket was an all-hands-on-deck, drop everything ticket. You could step in a bear trap, but still be on the hook for hobbling over to your workstation and fixing the ticket before you were allowed to gnaw your leg off. The issue was as vague as it was alarming: One of the clients that used the BBoM CMS system had all of their pages vanish.

Wonderful, Dan thought, our clients can't differentiate the difference between "edit" and "delete." Other developers joined him in researching the issue, and uncovered nothing. Even the records from the database were gone. After restoring from backup and poring over the code, it was resolved with a status of "not reproducible." Only one client had been affected by this so far, after all; there were plenty more that had no issues whatsoever. Still, Dan was troubled, so he took it upon himself to add a logging system to record whenever a "delete" was performed.

A few months went by, and the same thing happened. This time, Dan could check the log to find out the exact user ID and record ID that was being deleted, but to his dismay the log file was completely empty; no timestamps, no text, not even a newline. Dan quickly created a page and deleted it, checked the log again, and found that the logging was working fine:

01/25/2009 11:32:43 [Dan] Deleting page ID 108

The only way that a page could be deleted without it being logged is if something had direct database access (for example, all of the other developers on his team), or if someone had bypassed security. In reviewing his code, he noticed that it would only write to the log if there was a signed-in user carrying the action out. Still, the page was built to be inaccessible altogether if the user wasn't signed in, so that shouldn't have been the problem.

A Well-Intentioned Destructive Force

Cursing himself for not making the logging function record deletions in more detail or to log for non-signed in users, he went to the next logical place: the Apache logs. He knew when it was first noticed that the pages were missing, so he was at least able to narrow it down a little.

And that's when he saw something troubling.

208.77.188.166 example.com - [25/Jan/2009:08:29:00 +0100] 
  "GET /theCMS/pageEditor.php?action=deletePage&pageId=93 
  HTTP/1.0" 302 11605 "-" "ia_archiver"
208.77.188.166 example.com - [25/Jan/2009:08:29:24 +0100] 
  "GET /theCMS/pageEditor.php?action=deletePage&pageId=94
  HTTP/1.0" 302 11605 "-" "ia_archiver"
208.77.188.166 example.com - [25/Jan/2009:08:30:00 +0100] 
  "GET /theCMS/pageEditor.php?action=deletePage&pageId=95
  HTTP/1.0" 302 11605 "-" "ia_archiver"

ia_archiver is a bot that builds up data for Alexa's archive.org site, which takes snapshots of web pages so that you could see, for example, what AOL.com looked like in 1996. And Dan's company had been inadvertently hacked by Alexa's archiver! Or, perhaps, some brilliant and devious mastermind that was pretending to be the archiver. Dan went to check the validation code, and found the following:

if(!isset($_SESSION['usr_id']) || !isset($_SESSION['usr_name']))
{
    header('Location: index.php');
}

...Which is all well and good if the client respects the Location header. The crawler apparently doesn't, because it doesn't really have to. The bad thing is that any crawler, bot, or browser that can ignore headers could bypass all security on their site. Further, since deletions are performed via hyperlink and there's a whole column of hyperlinks where deleting something is literally a click away, it takes seconds (or less) for everything to be deleted permanently. Of course, this isn't the first time this has happened, and we've reported on a similar incident before.

Keep that in mind when designing your systems; use <input type="button" value="Delete Something" />, not <a href="delete.php?deleteWhat=everything&deleteMode=unrecoverable">index me!</a>. And maybe don't use just the Location header to secure your site.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!