• K (unregistered) in reply to Welbog
    Welbog:
    Leo:
    And what if the power goes down citywide ? Katrina-like ? Tnen obviously the off-site backup can't be on the same state. I suggest they set up the first ever Moon backup facility. They'd be covered even in the case of Earth's complete obliteration by an asteroid!
    But what if the moon crashes into the Earth? What if the sun goes nova? You have to be covered, man!

    Voyager contains the world's most secure music backup :-)

  • K (unregistered) in reply to Little Green Men, Inc.
    Little Green Men:
    We at Alpha Centauri will warehouse your data for you for a modest fee.

    That's not good enough. You know that the galaxy will collide with Andromeda some time in the future? I will not accept anything less than a multi-galaxy backup solution in at least 10 different universes.

  • AJS (unregistered)

    Why did they use MS SQL Server for the backup? FCOL, even if someone's been stupid enough to specify that the main database has to be Microsoft, you can always use a proper one for the backup. You only need a very minimal subset of SQL (DELETE, SELECT, INSERT and DROP) to make or restore a backup.

    Use something like Postgres (for reliability) or MySQL (for speed) for the backup database server, even if the main one is still an old McSoft one. And then get working on ditching the rest of the Microsoft stuff system-wide!

  • iMalc (unregistered)

    I don't suppose any of the backups were off-site?

    btw I'd rather listen to 'Paranoid' by 'Garbage', than whatever that other band mentioned was.

  • grep (unregistered) in reply to AJS
    AJS:
    Why did they use MS SQL Server for the backup? FCOL, even if someone's been stupid enough to specify that the main database has to be Microsoft, you can always use a proper one for the backup. You only need a very minimal subset of SQL (DELETE, SELECT, INSERT and DROP) to make or restore a backup.

    Use something like Postgres (for reliability) or MySQL (for speed) for the backup database server, even if the main one is still an old McSoft one. And then get working on ditching the rest of the Microsoft stuff system-wide!

    Yeah! Let's overthrow something!

  • Hatshepsut (unregistered) in reply to MyWillysWonka
    MyWillysWonka:
    ...then the microfilm would be set on a wooden table, have its picture taken which will then be scanned. OCR would then...

    Of course, as we learned in OMGWTF, 'OCR' is actually mouse-stroke recognition.

    Hence an army of data entry clerks would be employed to view the scanned pictures and transcribe the contents into the backup system by 'handwriting' with a mouse.

  • Dennis (unregistered)

    Just name the backup "Britney.Spears.Nude!!Great!!<date>.zip" and spread it via edonkey. Unless the whole internet breaks down, you will always have access to your backups!

    Bonus points if you forget to encrypt the backup first...

  • (cs)

    Reminds me of my favorite bash quote.

  • SteveC (unregistered)

    When I read the article, ironically enough, there were 60 quotes. I guess it's time to start the backup procedure ... again.

  • (cs)

    Ironically, 132 is the double bird of binary.

  • Akerbos (unregistered)

    Someone teach them (yummy) stochastic theory, please!

  • kastein (unregistered)

    wow. for once, a WTF is that someone is doing it TOO well, not badly.

    I think I approve of this guy's backup scheme! (just kidding, jesus. that's overkill to the nth degree)

  • Saemus Heaney (unregistered) in reply to AJS
    AJS:
    Why did they use MS SQL Server for the backup? FCOL, even if someone's been stupid enough to specify that the main database has to be Microsoft, you can always use a proper one for the backup. You only need a very minimal subset of SQL (DELETE, SELECT, INSERT and DROP) to make or restore a backup.

    Use something like Postgres (for reliability) or MySQL (for speed) for the backup database server, even if the main one is still an old McSoft one. And then get working on ditching the rest of the Microsoft stuff system-wide!

    I'm sorry, but what about this topic gave any reason at all for a Microsoft bashing?/ If MS was the issue then fine, but its not. pppftt.. And then you go and suggest swapping a orking MSSQL solution for a MySQL one?? I'm a fan of MySQL too, after having used both for many years, but they're still in a different ball park. Maybe you should have dropped the word Oracle in your post

    CAPTCHA: wigwam, because thats what AJS is.

  • (cs) in reply to Top Cod3r
    Top Cod3r:
    The real WTF is Ted doesn't seem to understand how backups work. At any one time I keep a full source code backup in a separate filesystem partiton hourly (my clients don't pay for me to re-do work, you see). I also have a keychain with 8 usb drives attached to "backup the backup" one for each hour of the day I work, except lunch time.
    Hourly backups? Sounds more like someone is not using a version management system, if you ask me. That would be the only reason a daily backup of the source code repository would not be sufficient. And do your clients know that they are paying you for wasting a full work day every few weeks doing manual hourly backups, just in case you lose a day's work every 6 months or so? No, I'd say it's not Ted whose understanding is lacking...
  • JM (unregistered)

    Slightly off topic, but I've discovered that attitudes (and methods) to backup and disaster recovery vary according to culture (ie. expectations of risk):-

    • in Amsterdam they really only care about floods. Which is interesting because the dykes haven't failed in several hundred years but planes have fallen on buildings in living memory (and killed over 200 people in the process if I remember correctly),

    • in Tokyo they don't really care that much about complete destruction of the building in earthquakes because "come the big one, there won't be a business left anyway". It's small failures like disks and CPU's they get anal about.

    • in London financial markets there is a very real "whack in something best practice and forget about it" approach. I learn't this on 9/11 when our ops people rang the NY guys and told them to basically "email us your mobile numbers, chuck your positions in a spreadsheet, copy them to laptops, desktops, whatever you have; get out of the building and we'll handle it from here - just forget about your disaster recovery, we'll handle it, you get yourselves out"

  • (cs) in reply to JM

    In the USA, its very common to see wackiness at small businesses. Their backup strategies apparently stem directly from FUD-created from how many times they've really had to do a recover. 0 = no backup strategy, and they'll be damned to start one until... 1 = HDD backup copy. 2 = HDD backup copy, and a CD backup at home. 3+ = wacky strategy like the one in this article.

  • (cs) in reply to iMalc
    iMalc:
    btw I'd rather listen to 'Paranoid' by 'Garbage', than whatever that other band mentioned was.

    "Dear Lord, I know I haven't talked to you lately. But it's time for me to ask and be given. I hope that commentary above was sarcasm, otherwise do as a favor and take him with you. Amen."

  • (cs) in reply to JM
    JM:
    Slightly off topic, but I've discovered that attitudes (and methods) to backup and disaster recovery vary according to culture (ie. expectations of risk):-
    • in Amsterdam they really only care about floods. Which is interesting because the dykes haven't failed in several hundred years but planes have fallen on buildings in living memory (and killed over 200 people in the process if I remember correctly),
    <SNIP>

    The North Sea Flood of 1953 was some 50 odd years ago and killed over 1800 people (though not in Amsterdam). Planes have fallen on buildings, but I don't think you need to devise a seperate backup strategy for that.

  • (cs) in reply to Beowulff
    Beowulff:
    Top Cod3r:
    The real WTF is Ted doesn't seem to understand how backups work. At any one time I keep a full source code backup in a separate filesystem partiton hourly (my clients don't pay for me to re-do work, you see). I also have a keychain with 8 usb drives attached to "backup the backup" one for each hour of the day I work, except lunch time.
    Hourly backups? Sounds more like someone is not using a version management system, if you ask me. That would be the only reason a daily backup of the source code repository would not be sufficient. And do your clients know that they are paying you for wasting a full work day every few weeks doing manual hourly backups, just in case you lose a day's work every 6 months or so? No, I'd say it's not Ted whose understanding is lacking...

    Whoooooooooooooooooosh...

  • (cs) in reply to fatdog
    fatdog:
    iMalc:
    btw I'd rather listen to 'Paranoid' by 'Garbage', than whatever that other band mentioned was.

    "Dear Lord, I know I haven't talked to you lately. But it's time for me to ask and be given. I hope that commentary above was sarcasm, otherwise do as a favor and take him with you. Amen."

    Seconded
  • (cs)

    I was thinking they get the guy who can write the smallest, and have him manually copy the entire database onto Shrinky-Dinks. Then, a short time in the oven later, PRESTO! Really awesome data compression!

  • Andrew (unregistered)

    They forgot green bar print backups. OMG!

  • Juanma (unregistered) in reply to Adam
    Adam:
    I worked at a company where the offsite backups were kept in a salt mine in a mountain, there was a company that managed storage there. At a disaster recover meeting, one team member was concerned that in the event of an earthquake we wouldn't be able to retrieve the backup tapes because all of the helicopters would be used for rescue work. We told him not to worry, none of us would be there to restore the tapes in that case.

    Well, either you are kidding (it sounds both like yes and not) or this one is 2 dimensions bigger WTF than the article's original topic. In a salt mine in a mountain??? But, who protects the backups from the Balrog?

  • StarControl2Rocks (unregistered) in reply to Marc

    All you have to do is provide us with the current location of the Chenjesu's famed sun device.

    And 4x10E47 backup copies of that location.

  • (cs) in reply to Juanma
    Juanma:
    Adam:
    I worked at a company where the offsite backups were kept in a salt mine in a mountain, there was a company that managed storage there. At a disaster recover meeting, one team member was concerned that in the event of an earthquake we wouldn't be able to retrieve the backup tapes because all of the helicopters would be used for rescue work. We told him not to worry, none of us would be there to restore the tapes in that case.

    Well, either you are kidding (it sounds both like yes and not) or this one is 2 dimensions bigger WTF than the article's original topic. In a salt mine in a mountain??? But, who protects the backups from the Balrog?

    A computer security professor once told me that to exchange public keys with a particular person, he'd had to go deep into a coal mine outside Cardiff. I'm sure this forum could quickly degenerate into a discussion of the merits of coal mines vs. salt mines for security purposes.

  • Joe (unregistered) in reply to sobachatina

    It indicates that all the overhead of the redundancy and backups leads to significant performance problems.

    Of course, the other way to look at it would be your interpretation. It could be that there are so few transactions on the db that the backups are very small.

    I think both are reasonable interpretations.

  • Joe (unregistered) in reply to sobachatina
    sobachatina:
    Pardon my ignorance but I'm a programmer (who doesn't write database code) not a DBA.

    What is the significance of the 60 queries? Does it just emphasize the overkill of the solution because there are so few changes?

    It indicates that all the overhead of the redundancy and backups leads to significant performance problems.

    Of course, the other way to look at it would be your interpretation. It could be that there are so few transactions on the db that the backups are very small.

    I think both are reasonable interpretations.

  • Madman1969 (unregistered) in reply to MyWillysWonka

    You have a evil twisted mind !

  • mathew (unregistered) in reply to Ted :(
    Ted :(:
    The good news is that most of these 'backup' processes have been eliminated, but my job's only half done.
    If you haven't already, take a look at rdiff-backup.

    I use it to keep 6 months of daily snapshots of our databases, on a server in a different city.

    A full backup is 13G, and is done monthly. The snapshots are only a few hundred KB each.

    I've used the backups once so far, to fetch an N day old copy of a database and retrieve data that some idiot had deleted using old fashioned cut and paste.

  • Ted :( (unregistered) in reply to Joe
    Joe:
    sobachatina:
    Pardon my ignorance but I'm a programmer (who doesn't write database code) not a DBA.

    What is the significance of the 60 queries? Does it just emphasize the overkill of the solution because there are so few changes?

    It indicates that all the overhead of the redundancy and backups leads to significant performance problems.

    Of course, the other way to look at it would be your interpretation. It could be that there are so few transactions on the db that the backups are very small.

    I think both are reasonable interpretations.

    In this case, I was trying to capture the effort expended to capture a delta that could easily be recovered in a matter of hours by a typist of average capability re-entering all data from a day's worth of updates.

    Basically the company has spent >$30K to develop a backup system that, even if we went down to weekly backups and lost all of our data the day before the next backup, could be replicated with no more than a days effort by a handful of office staff.

    For coders without DBA experience, 60 updates an hour is usually fairly trivial (especially when those updates touch single records or very small groups of records).

    For those without backup experience, doing nightly fulls (while in this case, if just one copy of the db is being backed up, is't that big a deal due to the overall DB size being <3Gb ) is usually not advisable because you're duplicating a lot of data that doesn't change. Incrementals or differentials should be done hourly/nightly/weekly (or other based on your own needs/requirements) while fulls should be done weekly or monthly (again, YMMV depending on how much of your data actually changes and how many tapes you feel like loading to execute a recovery).

    Either way, having 132 DB copies to capture a weeks worth of data that might approach 10Mb wotrh of changes is pretty retarded no matter what your data retention policy. It's the functional equivalent of doing a complete backup every 1.25 hours, every day, every week.

    And it makes my head hurt every time I think about it.

  • (cs) in reply to PSWorx
    PSWorx:
    They need another backup server. In space. In case some aliens blow up the planet to make room for an interstellar highway.
    I propose Deep Thought...
  • AdT (unregistered) in reply to Top Cod3r
    Top Cod3r:
    The real WTF is Ted doesn't seem to understand how backups work. At any one time I keep a full source code backup in a separate filesystem partiton hourly (my clients don't pay for me to re-do work, you see).

    The real WTF is that you store the backup on a different partition of the same drive and then accuse others of not understanding how backups work.

    PS: I bet those "USB drives" are flash sticks with FAT filesystems. Oh, and one voltage spike due to lightning or a crappy power supply will probably fry your HDD's built-in controller and any attached USB devices. Also, if you spend just 20 secs an hour changing USB drives and copying files over, then you should better have one failure that requires a restore every 90 hours, otherwise the time expenditure for the backups won't pay off on average. Like your customer pays you to write code, not play the stick jockey.

    Captcha: darwin (let him sort you out)

  • Darien H (unregistered) in reply to mathew
    mathew:
    If you haven't already, take a look at rdiff-backup.

    I use it to keep 6 months of daily snapshots of our databases, on a server in a different city.

    If you're doing directory-level backups, I assume you're shutting down the DB server (or otherwise flushing to disk) for integrity before doing the backup?

    At my company I ended up rolling a python script that would iterate through the databases on a system, doing a textual dump (pg_dump), and keeping diffs (xdelta3) for each day since Sunday. Since we have lots and lots of databases most of which change very little (hosting stuff), it seems to work pretty well, trading downtime for slower-than-normal time.

    Note that the default Unix 'diff' tends to have some nasty memory usage problems when generating patches for large files.

  • Darien H (unregistered) in reply to StarControl2Rocks
    StarControl2Rocks:
    All you have to do is provide us with the current location of the Chenjesu's famed sun device.

    And 4x10E47 backup copies of that location.

    And of course, your data will be fully encrypted for safety, with myself as the possessor of the sole quantum encryption key unit. How do you know it will be safe? In a nutshell, I guarantee it with my life, for it is linked to my own well being, since I keep it under my pillow.

    Captcha: Smile.

  • Zygo (unregistered) in reply to Ted :(
    Ted :(:
    BTW, great idea on the extra-solar back-up solution! I'll propose it to the Board at their next meeting... never know when our Sun's going to go Red-Giant ;)

    I'm a software developer, not an astronomer, but I've heard some rumors...

    1. Every few dozen million years some really large object smacks into the Earth, wiping out data repositories worldwide. This happens much more often than the Sun becoming a red giant, so your backup strategy should definitely include measures against this much more likely event.

    2. A supernova occurring anywhere within a 50 light year radius will probably fry your database server and your IT staff. Need backups for those too. Consider outsourcing to a reputable customer-service-oriented alien civilization.

    3. The Milky Way galaxy is on a collision course with Andromeda. They'll collide in a billion years or so. Now would be a good time to start looking for strategic off-site locations, before the rush inflates real estate prices to levels that are, well, astronomical.

  • Zygo (unregistered) in reply to chran
    chran:
    No, they need to build a large radio-telescope, like Arecibo.

    Then, every night they send out their backup! Restoring is a piece of cake! You just send out a space ship to the front of the data, and then you ...

    Well, if you ever need to restore, FIRST you need to invent a space ship that travels faster than light!

    Nah, you just need a space ship that is located at the right distance for your data retention policy. So, e.g. if you keep your data for two years, you can park your ship in orbit around Vega.

    Recovery time is a bitch though.

  • James (unregistered) in reply to Marc
    Marc:
    Darien H:

    How much does it cost to find out why your bridge just turned purple?

    You each get +400,000 points for the reference. Seriously, Darien, you made my day.

    Any other fans out there, check out Ur Quan Masters, a complete rewrite of the engine using original game content (which I believe is available as abandonware on the internets).

  • deadimp (unregistered)

    Why do you people talk of data backup in space? That's just insane, completely improbable, redundant, and illogical. Instead, you can work with something better, and a little more viable. Why not invest a bit of money into time travel and just store the backups in the past? It's not that far away, maybe just a couple of steps in our 3rd dimension. Nothing happens there that already hasn't, so you're good unless you were screwed! Plus, since all future backups are now at one point in time, you no longer have to run your server, because everything's already happened... unless it didn't because you thought it did which caused it not to so it ended up not happening because it actually did.

  • Brad (unregistered) in reply to StyxRiver
    StyxRiver:
    I've been reading TDWTF for a while now.... I think this is the first time that part of me died while reading this...

    If you've been reading this for awhile, and that only happened just now, something is SERIOUSLY wrong with you.

    sigh

  • Mircea (unregistered) in reply to sobachatina

    It actually means "Hey, 60 queries? Maybe we should backup hourly...". Got it?! :)

  • Mircea (unregistered) in reply to sobachatina

    My above comment refers to:

    sobachatina:
    Pardon my ignorance but I'm a programmer (who doesn't write database code) not a DBA.

    What is the significance of the 60 queries? Does it just emphasize the overkill of the solution because there are so few changes?

  • Workin'DBA (unregistered) in reply to Ted :(

    I'm curious to know how often the logs are backed up. At this rate or recovery, anything more than 5 minutes would be gastly!

    a/c

    CAPTCHA:dreadlocks (deadlocks would have been funnier)

  • bad coder (unregistered) in reply to deadimp
    deadimp:
    Plus, since all future backups are now at one point in time, you no longer have to run your server, because everything's already happened... unless it didn't because you thought it did which caused it not to so it ended up not happening because it actually did.

    Exelent.... Why would we do anything else?

  • (cs) in reply to Ted :(
    Ted :(:
    In this case, I was trying to capture the effort expended to capture a delta that could easily be recovered in a matter of hours by a typist of average capability re-entering all data from a day's worth of updates.
    In many businesses there is no paper copy. Think of a telephone center taking orders. Or a website taking orders.
  • Hatshepsut (unregistered) in reply to Someone You Know
    Someone You Know:
    A computer security professor once told me that to exchange public keys with a particular person, he'd had to go deep into a coal mine outside Cardiff.

    Why would a Welsh coal miner want to exchange public keys with a computer security professor anyway?

  • Ted :( (unregistered) in reply to Tony Toews
    Tony Toews:
    In many businesses there is no paper copy. Think of a telephone center taking orders. Or a website taking orders.
    I definitely agree with your point, and those would be examples of reasonable grounds to have multiple copies of data staged at near-intervals (through diffs, incrementals, mirrors, etc.). As in my 'WTF', our DB doesn't record these types of transactions, though. Anything related to $$ is actually processed outside of our DB then entered into the DB by our staff.

    In our business, I'd say there's a paper copy (or a least a paper-trail, indicating what should have been changed) for about 80% of the DB's transactions. The remaining 20% could be recreated by sending an e-mail to our mailing list telling them that any changes they made to their account over period 'x' should be resubmitted (very amateurish, but not the end of the world, either).

    BTW, thank you all for your alternative backup suggestions. They are very much making my day. I'm currently working on a quantum-based, phased, backup array which elevates matter into an alternate dimension in the event that our entire existence is obliterated due to some presently unknown catostrophic cosmic force. I'm thinking restores are going to be difficult if an event occurs, but hey, at least the data will be there for some alternate-dimension being to retrieve should the need arise. :)

  • (cs) in reply to Ted :(
    Ted :(:
    Tony Toews:
    In many businesses there is no paper copy. Think of a telephone center taking orders. Or a website taking orders.
    I definitely agree with your point, and those would be examples of reasonable grounds to have multiple copies of data staged at near-intervals (through diffs, incrementals, mirrors, etc.). As in my 'WTF', our DB doesn't record these types of transactions, though. Anything related to $$ is actually processed outside of our DB then entered into the DB by our staff.

    In our business, I'd say there's a paper copy (or a least a paper-trail, indicating what should have been changed) for about 80% of the DB's transactions. The remaining 20% could be recreated by sending an e-mail to our mailing list telling them that any changes they made to their account over period 'x' should be resubmitted (very amateurish, but not the end of the world, either).

    BTW, thank you all for your alternative backup suggestions. They are very much making my day. I'm currently working on a quantum-based, phased, backup array which elevates matter into an alternate dimension in the event that our entire existence is obliterated due to some presently unknown catostrophic cosmic force. I'm thinking restores are going to be difficult if an event occurs, but hey, at least the data will be there for some alternate-dimension being to retrieve should the need arise. :)

    Well, I don't know about anyone else, but this is one of the very few times that I've seen a solution that definitely should not be tested before being put into production...

  • (cs) in reply to Darien H

    Throw off the yokes of your Ur-Quan masters, Darien. :)

  • Frenchier than thou (unregistered) in reply to Brad
    Brad:
    StyxRiver:
    I've been reading TDWTF for a while now.... I think this is the first time that part of me died while reading this...

    If you've been reading this for awhile, and that only happened just now, something is SERIOUSLY wrong with you.

    sigh

    He was in denial before :-)

  • Kristopher (unregistered) in reply to Mircea
    Mircea:
    It actually means "Hey, 60 queries? Maybe we should backup hourly...". Got it?! :)

    At that rate, they could have the DBA changes incrementally chiseled into granite.

    Granite tablets are flame-proof, and do not suffer bit-rot.

Leave a comment on “A Backup's Backup's Backup”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article