Call me a Skeptical Sally (actually, don't), but whenever I hear someone complaining of random file corruption, I don't really believe them. Of course, it's a wonderful excuse if you don't know why your code doesn't work or you just slacked off and didn't get some Word document done; maybe you've even used it a few times. Still, that doesn't change the fact that random file corruption rarely never happens.
File corruption that results from inexperience (or poor training) is the real danger. Today's story comes to us anonymously, and is about an IT guy that we'll call Ness. Ness got a call from a client, who was justifiably concerned about some major problems they were experiencing with their production database. Ness did some digging and found that the database was so corrupt that the only option he could think of was restoring from the most recent backup.
Ness reported his findings to the backup administrator, who wanted to do some of his own research and call back. Ness was starting to get a little nervous, as this downtime was hurting the company; several employees were sitting around unable to work, and Ness wasn't even sure how far back the corruption extended. Just a few minutes later, the administrator called back, reporting in a somber tone that the most recent backups of the database were corrupt, too, and couldn't be restored.
Ness was granted access to the backup server by the desperate administrator and figured out precisely why restoring the database was impossible. Backups were stored in 2GB chunks and copied to tapes. The streamer tapes had gone bad, and higher-ups decided that the cost to buy new tapes wasn't justifiable. After all, they'd just bought a new server that had plenty of free space on the hard drive. It was determined that it'd be cheaper to hire an intern to upload the files via FTP daily.
This backup procedure had been in use for about a month, which was also around the time that the last usable backup was created. Ness was surprised to see that the backups were several MB less than the expected 2GB, as well. He was puzzled by this, until it occurred to him.
"Well," Ness asked the administrator, "did anyone tell the intern to type 'BIN' before uploading the files?"
"No, what does that do?"
For those unfamiliar, files are sent in either binary or ASCII. BIN tells the ftp client that the next file(s) should be sent in binary so that character sets aren't synchronized between client and server, and newline characters aren't replaced for the target system. Since the backups were uploaded in ASCII, data in the file changed as a result of the upload, making the files impossible to restore from.
Ultimately, the month-old backup was restored, and after a few weeks things were sort of back to normal. After this incident, management approved the purchase of enough tapes for the next few years of backups.