- Feature Articles
- CodeSOD
- Error'd
-
Forums
-
Other Articles
- Random Article
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
Don't these tapes have a limited number of uses? I hope not :P Well, I think there have been worse errors in the history of mankind, but it still is funny.
Admin
There's a new, high-tech thing they've come up with now to help prevent that. It's called a label.
Admin
Wow. Tapes.
In this day and age of DVDs and external terabyte drives, who would use slow, sequential, cross-system incompatible devices like tapes for their important data backup?
Well... kudos for attempting to backup at least.
Admin
This is ultimately Marcel's fault because two important rules of sytems have been violated:
1.) Test backups with practice restores periodically. How else can anyone be confident in the backups?
2.) And never, NEVER underestimate the stupidity of users, or in this case, operators.
Admin
Regardless of the backup media of choice, it's important to test that the backups work. That means restoring the data onto a (hopefully identical) system and turning it on. And making sure that all the data you want backed up is actually on the backup media.
The company I used to work at employed a person who's job it was to make the backups (and do other systems work, but mostly backups). I don't know how many tens of thousands of dollars we paid him to MAKE the backups. Apparently we never paid him to actually CHECK the backups.
When it came time to restore an important system (the one that had our source code control repository!) it turned out that that system was not backed up. It would have taken maybe 2 minutes of time somewhere to check that the machines supposedly backed up were actually being backed up.
Admin
And Chris said: "Well, on Tuesday mornings I grab the WTF entry from the top of the SQL query in the 3rd floor table, make a backup, and put it in the daily homepage for the site."
captcha: riaa - I almost had a heart attack
Admin
You would be surprised at the number of tape backups that are in place. My company recently decided to backup some servers "at long last" and they are done with a tape based system. If it ain't broke don't fix it.
Admin
You're kidding, right?
Who in their right mind would use DVDs for backup? External hard drives? Better, but "moving parts" in your backup media isn't the best solution.
Tapes are still the best option in terms of cost, speed, capacity, and shelf life. Even in a big "disk to disk" backup system, the data is probably going to end up on a tape eventually.
Admin
Admin
This could be called the "Ping-Pong Backup Strategy".
It is the recommended backup strategy for personal data like dialled telephone numbers on a network operator's mainframe, or requested IP-numbers at ISP computers.
'tis for privacy reasons, in case anyone wonders :-)
Admin
Why do you think it got an error? Tapes are often limited to 200 - 50000 writes. What lasts an eternity if you cycle them, but may last only 6 months if you use the same tape every day ;)
Admin
First, tapes have a long history, are (relatively) cheep, and have high capacity. We pay $15 for a 200 GByte tape (compressed, 100 GByte uncompressed). Where do you get a hard drive that cheap? Or DVDs that convenient? It's the right tool, sometimes, some places. Especially if you have to have 100 on the shelf. (Don't ask!)
But it hasn't always been that way. When I started, the rule on the dev box was to leave the tape in the drive. If it ejected, it was because the backup failed. !!!! The production server ejected no matter what so the backup guy could pull it out & put a note on it. The two servers, with different tape drives, had no formal plan. The main production server had post-it notes for monday-friday. Recovery mostly didn't work.
Sucked.
Now, a plan. Even testing. We were lucky, though. Since then, because we test & use a plan, we've been able to recover a lot of data and saved at least a couple of jobs.
Admin
A tape has better shelf life than a DVD? Now you're the one telling jokes.
A tape breaks, molds, erases in magnetic fields, and must be kept within relatively strict environmental parameters. On the other hand, a dormant DVD has a relatively infinite shelf life. And they are dirt cheap. And they have no moving parts, unlike tape cartridges. Tapes are probably one of the worst storage media choices these days.
Admin
Anyone backing up large volumes of data. Last time I checked, the tradeoff point between tape and hard drives was at 5 TB of data: once you need to keep more than five terabytes of data in your backup set (which typically corresponds to 1 TB of data on the computer), it's cheaper to use tape than to use hard drives.
DVDs are almost never worth using. They're cheap per gigabyte, but they just don't hold enough data.
Admin
Right, but who's got an autoburner for 100 DVDRDLs spindles?
You can either use a single DLT-S4 with 800GB per cartridge at 60MB/sec..... or you can use a 100 DVD+R-DLs spindle at....how fast do these burn now? 16x? (that's like what? 22MB/sec?)
Admin
Well, I do.
DVDs are slow, single use (yes, I know about DVD-RAM, costlier and I don't trust them) and don't hold much. I put 180 GB onto a single DLT tape every week. That would be 40 DVDs! So, to use DVDs, I would need a DVD library device and deal with unloading, labeling and loading 40 DVDs every Monday. No thanks!
Hard disks are good, my daily backup rotation goes to disk, but I need to send backups off site. I transfer crucial data to the DR site over the wire nightly but sending everything would take too long. I don't want to be plugging and unplugging devices to send them to storage.
Also, I want to keep a rotation so that I can restore data from 2 weeks or 6 months ago. That's hard to do with hard disks and gets pretty expensive if you want to keep a full rotation going.
I'll keep using my tapes with restore checks.
By the way, most backup software can be set so that it won't overwrite a tape until its turn in the rotation. That simple setting would have helped with this WTF too. The bottom line is that not testing your backups turns them into a WOM (Write Only Memory) process. They will not be readable when you need them.
Admin
So the only reason they had a backup, the six-month-old one, was because six months ago, one of them was away and forgot to do their job?
Admin
It doesnt sound like shelf life is an issue. A two year old record of who has what books checked out probably isnt all that useful. If they need to restore from backup, they'll need a recent backup.
Tapes might be good for long term archiving of data that you no longer need immediate access to, but DVDs are what you want for short term backups. Plus they take up a heck of a lot less space.
Admin
Or at least look at the dang things to be sure they are up to date!
Admin
Possible compromise: Use tapes for your full backups, say once a week, and DVDs for daily incremental backups. 1 Tape + 7 DVDs is prolly cheaper than 7 tapes. Although if you only have about say 21 tapes and cycle them every 3 weeks that wouldn't matter much. Downside: pain in the arse to restore from incremental backups.
Hmm, no fsck it, tapes are just better.
Admin
Besides their unsuitability for large amounts of backup, DVDs require controlled environment, careful handling and good quality media to achieve long storage life. Ref
Frankly they haven't been around long enough to say with certainty how long their life will be.
Admin
Ok, maybe i'm retarded, or don't understand what exactly they are backing up, but how would that scenario result in a backup of more than a day old?
Ex.
Monday 10/8/2007 Backed up to Tape 1. Tuesday 10/9/2007 Backed up to Tape 1. Monday 10/15/2007 Backed up to Tape 1.
Would that not just overwrite with the current day's backup each time? or does it not perform the back up if the tape is used?
Admin
Would have been a decent enough process if they had used a queue instead of a stack.
Admin
I'm assuming Chris was hired the day after that 6 month old backup. From that point forward, they were writing to the same tape. All that wear and tear is why that tape crapped out after 10%, so the second tape on the stack was 6 months old.
Admin
Now to see who forgot to back it up (to reward that person, not punish), was the functioning tape from the basement or the third floor safe? Sounds like a good logic puzzle.
Admin
Someone who cares whether his data will still be readable in 10 years.
Admin
40Gb worth of content would use about 9 DVDs or so, which is about the thickness of a 400Gb tape (and tapes are square, unlike DVDs, which means they can be packed more easily. Plus, they don't need jewel cases to keep them from getting scratched.)
Admin
Why not Jan's?
Admin
Somebody must have missed a day, some six months ago, which caused the other one to go one step deeper into the backup tape stack.
Admin
Admin
Tape rocks over DVD. I mean seriously, 4.7GB vs up to 1.6TB, yes that's TeraBytes per cartridge. Even Magneto-Optical drives and libraries are starting to fall out of the market, because they simply don't have the storage capacity or transfer rate of tape.
Burnable CD's/DVD's have a shelf life of about 10-15 years in an ideal environment. Linear tape media (DLT, SDLT, LTO) about 30 to 50 years. The newer media, SDLT600 and LTO-3, are extremely hard to degauss. Degaussers that are rated for those type cost many many wampum. Which is why the boss hasn't ordered ours, yet.
Also, an LTO-3 runs at a sustained 70-75MB/sec (real-world). The new LTO-4's are probably close to 160MB/sec. HDD's can't pull that kind of sustained transfer rate, which is why you should be backing up off of arrays.
All this reminds me of "Live Free or Die Hard". You know, where the bad guys are backing up 500TB of data on to "a couple of hard drives". A couple? Try a thousand. You would need five 42U cabinets filled completely with 14 Arrays with 14 500GB hard drives each. And that doesn't even include power distribution/protection, controllers and FC switches to stitch it all together.
See, what they needed was a STK L700. 696 slots and 20 LTO-3 drives. If my after movie calculations were correct, you could have the entire half petabyte backed up in about 10 hours. If they used LTO-4's they wouldn't need the expansion module (so you could get really cool shots of the robotics through the window) and it would take about 5 hours. Assuming they could get the raw bandwidth off the SAN.
Trust me, tape rocks. Especially when it pays the bills. ;-P
Admin
Size, pure unadulterated size is why people use tape. Our backup program consists of 3 sets of daily backups rotated weekly: Week 1 is the live set & is stored in the firesafe. Week 2 is last weeks & is stored offsite (MGR house in a firesafe). Week 3 is 2 weeks old & is stored offsite (consultants office in a firesafe). For us that's 15 40GB tapes - in terms of actual DVD usage, it would turn out to about 15 sets of 5-6 DVDs each. I would need a rack of DVD burners to pull it off. For people who are using the 100MB+ tapes, it's just insane to suggest that a DVD is a suitable medium. I suppose I could do it with hardrives, but popping them into the firesafe everynight is going to be hell on the connectors.
Admin
Admin
Almost, I think someone just slipped up back then.
Also, The whole plan with the tapes was to cylce the whole stack of 180 tapes (three months) around a few times, and throw them away after XX years. It also seems that, for some reason I can't grasp, the libarary is required, by law, to keep information five years back. Then it's more usefull to be able to dump it all on a tape. 800gb does not sit well on DVD's.
The most horrible thing were all the angry people, who didn't really the appreciate the whole story and were very confused why they couldn't check what they have at home, or expand the period.
You can't really tell people you've lost all records and they're totally clear to never return the books...
Admin
Looks at watch Looks at DVD Looks at watch
Give me a little bit and I'll let you know.
Admin
pebkac errors are one fo the best ... but I totally agree, and as the article said they seem to have a solution that might solve their problem.
Admin
pebkac
Admin
In Theory of Computer Science they teach about countably infinite sets. Maybe that's what he was thinking about? What's next, he's going to tell us DVDs > Church-Turing Thesis.
What's his favorite theory? The Pumping Lemma? Are DVDs > Pumping Lemma?
http://en.wikipedia.org/wiki/Pumping_lemma
Admin
No. There the problem exists on the keyboard.
Admin
And they never noticed that the stack(s) of dvds always had the same height? (I mean, if I took a dvd from one stack and put it on another stack each day, I would expect the first stack to slowly dissapear... :D)
Admin
SenorLapiz is the real WTF. We're not talking about backing up PHPbb on your Ubuntu "server" (an old Dell in your mom's basement).
Admin
Write-able discs will degrade in heat and humidity. So when I lived in South Florida, discs that I burned would last about 12-15 months before they became unreadable. Tapes won't die if you leave them in 100% humidity 80F for years. Burnable discs - even the archival ones - will anneal and slowly erase in that environment.
If you want recordable CDs and DVDs to last decades, you need to store them in a fridge.
Now. That being said, one company I worked for, I set up a tape rotation scheme and a simple 1-page description of how the rotation worked. The guy who did it after I quit, thought that was too complicated, so he left the tape in the drive and used it every night. Things were a mess, so they lost the back up tapes as well. When the computer needed to be restored, we discovered that the oxide layer had been worn off in many spots, leaving the tape material to look like unsticky scotch tape. When this happened, the "restore" of the production database was when I pulled a year-old development copy that I had at home - they had already reformatted my old development computer - and that year-old copy was the latest copy.
Admin
Yeah, I have somewhat lower reqs, and DDS3 tapes run about $5 ea. for 12G uncompressed, which is plenty. Is $50/yr worth having your data after a crash?
Admin
Marcel should have checked the next tape on the bottom floor. The chances of it containing a recent backup are the same as the next tape on the 3rd floor, which he did check.
If Jan was off sick one day, Chris would have moved another tape from the 3rd floor onto the bottom floor and the bottom floor would then contain two recent backups. One would get overwritten when Jan comes back the next day, though.
Admin
Under the best storage conditions, their shelf life may be as long as a tape. We won't really know that for years.
I assume you are talking about DVD+R. The error detection and correction on DVD-R is laughable.
As far as I'm concerned, DVDs as backup are useful for those organizations who either don't value their data or who have clueless management who have no idea what backups are.
Hmmm. That describes where I work perfectly. We use DVDs for backups precisely because the president of the company would rather spend the cost of a decent backup drive and tapes on useless toys instead.
Admin
Brillant!
Admin
Errrm, less space? How? A 400GB LTO cartridge is no wider than three or four DVDs (in cases, which I assume you'd want to keep your backups in), and is neither taller nor deeper.
400GB would take what, 90 DVDs? That's something like 6 or 7 inches of just the DVDs. If you use slim cases, those are <a rel="nofollow" href="http://www.cmcdisc.com/sjc.htm"" target="_blank" title="http://www.cmcdisc.com/sjc.htm"">5.15–5.30mm, so 1.5 feet of shelf space, vs. around 1in. for the tape.
It takes somewhere around 20 minutes to burn a full DVD, so that'd be around 1¼ days strait to write, another 1¼ days to verify. Maybe with a 16x drive and very good media (there went the cheap...), you could get that down to the write and the verify in a day and a quarter (10 minutes/disc). The tape run would take maybe 10 hours for write & verify (or less; many tape drives do both at once).
Make sure to compare the failure rate for 90 DVDs vs. 1 tape.
Also, when pricing it out, remember that the 1 tape is reusable, the 90 DVDs are not. Oh, and include the price of the cases in your DVD cost — tapes always come with a case, AFAIK.
If and only if your backup set easily fits on one DVD (maybe two) should you consider using DVDs for backup.
Admin
He was probably thinking of VHS tapes.
Admin
"cross-system incompatible"
They don't work in his beta deck.
Admin
Welcome to the wonderful world on transfinites...