- Feature Articles
- CodeSOD
- Error'd
-
Forums
-
Other Articles
- Random Article
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
How does one pour a cup of air into a cup of coffee and replace the coffee? Must be some pretty heavy air.
Admin
Requires some unusual circumstances it appears, but it is possible for a gas to displace a liquid under 'normal' gravity. (More.)
Admin
Well you only have backups if you have actually proved that you can restore..... If not, the only thing you have is some expensive toys (tape backup robot etc)...
(and that's why i personally once every few months, renames a random (not used) file on one of our data logging servers and restore the original and then do a diff to make sure that the restored file and the original are equal! )
Admin
So let me get this. TRWTF is that both Dick and Q-Max got bigger over the years? Is this because Dick is being paid too much pizza and doughnut money? Something needs to be done about that or he won't be able to get through the default gateway.
Admin
Definiitely possible to replace liquid with gas temporarily. It's called "bubbles". It is a household experiment that anyone can perform in the bath.
Admin
Admin
I wanted to write a very clever comment, but I'm going to check our backups instead.
Yes, we never restored too. Not even as an exercise :smile:
Admin
As soon as the writer introduced the direct-to-tape part, I immediately noticed that there was no mention of what was done with the copy-job, and it was clear that that would cause some sort of grief somewhere.
And it did!
Admin
Soo... the usual "we have backups but never tested restore".
Shouldn't be that hard to recover data from hard drives out of servers that got a little wet from above, no?
Admin
yeah.... as far as i'm concerned that's called "not having backups"
because i've never met a situation where someone had backups that they never tested restore and the restore actually worked in a DR situation.
Admin
Well, let me offer a counter-evidence. Recently I have apparently been the first one in our company to have ever asked to restore something from a backup, but even though it took us a while to find out how to access the backup at all, it did actually work. I guess an untested backup is still better than nothing at all.
But n=1 doesn't mean much, of course. I could also tell this anecdote where I lost a lot of data because I thought I had a back-up, only to discover that it had been rendered unusable when I sent it through FTP, and the FTP client thought it was a text file and needed newline conversion... So in general, I agree that testing is important.
Admin
I'm just thinking:
So the backup data has really only been unlinked, not overwritten. I bet it's still possible to recover.
Admin
Not if that happened before the data was written to the tape. If the empty directory was the only thing written to the tape, I don't think you're going to be able to recover the data that wasn't written.
Admin
That's not how I understand the article.
Admin
The article says "copy a backup directly to the tape library," but I would assume there's some buffering involved. I guess I was thinking the job would send all the data, then it would be written to tape, which would leave a window in which the data could be clobbered by the second, bogus backup job. If it's not buffering, or it's working one file at a time, then you're right.
Admin
I'm pretty sure this is how fertiliser irrigation works.
Admin
It should be recoverable. If you store an entire backup on the tape and then overwrite with an empty directory, it shouldn't erase the entire tape. It will just store a new backup (with an empty directory) at the start of the tape.
Software that can skip over that region and read the remaining blocks should be able to recover the files. I wouldn't expect a lot of data to be lost, but (depending on the backup software used) it might be a bit of a pain to recover the rest.
Admin
Agreed, but if it was overwritten on the backup machine before it wrote it to the tape, it might not bother writing the overwritten data to the tape in the first place. The article doesn't provide enough details to know for sure which of these happened.
Admin
First rule of IT disaster recovery: always test that your backups actually backup the data you want backed up.
Unfortunately most IT depts fail to follow this simple rule and when they actually need the backup they find it at worst useless, at best containing older data than what they need.
Admin
I would expect a few years of backups to be lost, as they were all erased before being written to tape. More if they were reusing the old tapes.
The article is short on technical details, but my interpretation of it was that the old backup process exported all of the data to a local filesystem, let's call it "server1:/export", and then left it there while a mirroring job copied it all to "server2:/import". Naturally this process would erase any files which already existed in server2:/import, because otherwise the backup size could grow without bounds. Once server2:/import contained a complete copy of the exported data, it was all written to tape and then immediately forgotten about.
The new process sent the exported data directly to server2:/import, leaving server1:/export empty since it was no longer needed. The mirror process then kicked in, making sure that server2:/import was an exact, empty, copy of server1:/export, and then that empty filesystem was happily backed up to tape.
For the data to be on the tape, it would have to have been copied directly there without ever landing on a disk. That's not impossible, but it would not only require that the "tape robot" on server2 be dedicated to the use of that single backup process. Since Disaster Pete went to all the trouble of implementing the original system it suggests that the tape library is probably used to handle more jobs than just the one, so caching to disk would be more likely.
As the tape library was described as "offsite", Dick might be able to try recovering data from server2:/import, but the now-free disk space would likely have been overwritten with data from other jobs by the time he figured out that he needed it.
So, um, yeah. Test your backup and restore strategy or get strangled by a Dick.
Admin
But the metadata about that empty backup would be written, plus possibly some extra random shit. If the tape was being used in such a way as to put each backup after the end of the last one with round-robin tricks to reclaim old ones as the tape ran out of space, with some time systematically erasing everything is entirely possible.
A backup whose restore is not tested properly is not a backup. It's just a security blanket.
Which reminds me, I've got some backups to test…
Admin
Actually not even that - it's a crumb you can throw at pointy headed bosses that don't know any better -- until they do.
Admin
Easy. Do it all upside down submerged in mineral oil.
Admin
Let's all recite with the AHole DBA: "Any untested backup is not a backup"
Usually one of the most expensive lessons to learn. A study showed: 1- That I am incredibly handsome (confirmed by my mom). 2- Businesses that lose over 80% of their data have gone bankrupt over 95% of the time within 5 years.
What small business owner would think his untested cloud backup is the demise to his dreams and aspirations?
Admin
I see where you're going with that idea, but mineral oil isn't going to do the trick. You need a liquid denser than dirty water; mineral oil is less dense, and the coffee will sink.
Admin
Lesson to learn: There are two types of people: Those that will be screwed by not having backups and those that have been screwed by not having backups (and have subsequently learned their lesson).
Admin
The air will be in the cup and the coffee will not. What's wrong with the solution?
Admin
Just that in this case, even if you test restore backups each day, it's not going to help.
(This backup is there when everything is okay, the backup is lost only if Oracle backup fails.)
So the actual lesson learnt from this story is that, you had better keep multiple days of backup. So that if the latest backup has anything goes wrong, at least you have the backup for the day before.
Admin
If, instead of a cup of coffee, you had a narrow-mouthed bottle, you might be able to "pour" air in faster than the coffee and mineral oil could exchange places on their own. In that case, the air could displace the coffee directly. But for an ordinary cup, I think the coffee would be pouring out before you could invert it to the point that it would trap the air. Unless maybe you sort of flung the cup over so that acceleration was forcing the coffee into the cup until it was inverted.
Alternative solution to the original problem: "Pour" the air into the cup very quickly. The air's momentum will apply a force to the surface of the coffee, pushing it aside and over the rim of the cup (and spraying it all over the room).
Admin
"pour a cup of air into" is not the same as "pour a cup of coffee out of"
I wrote that sentence because that's exactly what happens in the article: a directory that has data in it somehow has an empty directory copied over it. I don't know of any filesystem where copying a directory with the same name doesn't merge the directories. Pouring air into a cup of coffee should not do anything to the cup of coffee; copying an empty directory over any directory should be a no-op.
Admin
Did I say it was? No, I didn't. I was just explaining why @BaconBits's proposed solution was not actually a proper solution to the problem as stated; it too is pouring the coffee out. I also examined a couple of other proposed solutions that could work, but only by stretching the definitions of "pour" and/or "cup" to or beyond the breaking point.
Admin
Depends on the tape technology. Quarter Inch Cartridges (QIC tapes) have a strange way of recording: the data is written one track at a time, the first one forward, the second one backward as the tape is rewound, then forward, etc.
Erasing, on the other hand, is done by an erase head that covers all the tracks at once. When you write the first track, the erase head erases all the tracks just in front of the write head.
So you would lose the beginning of the first track, the end of the second, the beginning of the third, etc. Most of the data would still be readable, but the recovery is more complex because there are holes in the middle of your data.
Admin
OK, so use mercury instead. That's pretty dense.
Admin
Admin
Rather extreme, but yes, that would work. Something like bromomethane could work, too, if it was a cup of iced coffee (it boils a 4°C). Or iodomethane would work even with warm (not hot) coffee. Elemental Br2 has suitable density and boiling point, but poses certain difficulties in handling.
Indeed, but far from the densest. There are one or two users here who compete (un)favorably with iridium, and if pureed would be suitable for use in this experiment (except that they're probably miscible with coffee).Admin
I don't make coffee out of neutronium…
Admin
Admin
this reminds me of a story: an employee mis-interpreted the message on the disk, "disk must be formatted before use", and re-formatted EVERY disc before using it-literally the first thing after putting it in the drive...and his job was doing the backups...so when he tried to restore a backup...POOF!
Admin
Concordet sermo cum vita — Пусть речь соответствует жизни.