• (cs)

    That's almost one per minute!

  • (cs)

    And all 4 servers were in the same room, plugged into the same powerstrip...

  • (cs)

    To say, I used to be a paranoid about backups.

    BTW, while I was reading this WTF, my MP3 player began to play "Paranoid" by Black Sabbath. Whoa...

  • StyxRiver (unregistered) in reply to akatherder

    I've been reading TDWTF for a while now.... I think this is the first time that part of me died while reading this...

    CAPTCHA: cognac...probably Ted's new best friend

  • nerdierthanu (unregistered) in reply to akatherder

    They should really add an offsite backup, and extra hardware to restore to in case all the onsite backups fail.

    captcha : sanitarium

  • Look at me! I'm on the internets! (unregistered)

    I would have had IT staff do a printout of the entire database every night and fax it to a branch office where it could be converted to microfilm.

  • Dale Williams (unregistered)

    Hey, Better safe then sorry!

  • Ted :( (unregistered) in reply to Look at me! I'm on the internets!
    Look at me! I'm on the internets!:
    I would have had IT staff do a printout of the entire database every night and fax it to a branch office where it could be converted to microfilm.
    'Ted' here...

    Oh, if you only knew how close to reality that actually is... It turns out that in addition to all of the above, once a week there is an export of three tables in the DB to an XML file, a formatted HTML file and a printer-friendly version that is, once a month, actually printed. The printout takes 5 reams of paper (2500 sheets), the better part of an hour to complete and kills a small forest in the process.

    The good news is that most of these 'backup' processes have been eliminated, but my job's only half done.

    And to the user who noted that these are probably all in the same room, on the same power strip, you're 50% correct: same room, different UPS's... brought this up on my first day here and my Director simply said "yeah, I guess that wouldn't really help if there was a fire or something"

    There is currently a very large, head-shaped dent on my desk.

    /the gods hate me, my captcha was 'gotcha'

  • (cs)

    We have unique backup system. Every time anyone or anything posts a change to the database, the whole thing gets backed up. To make restores easy, the backup is stored in the same database. I learned this on my first day, when someone said to me, "We have a unique backup system...." --Rank

  • Mike (unregistered) in reply to Ted :(

    Somebody buy that poor man a beer.

  • diaphanein (unregistered) in reply to StyxRiver
    StyxRiver:
    I've been reading TDWTF for a while now.... I think this is the first time that part of me died while reading this...

    CAPTCHA: cognac...probably Ted's new best friend

    Eventually it happens to us all. Welcome to the distinguished club.

  • anon (unregistered)

    Screw DBAs. They deserve to suffer. They only exist to prevent developers from getting work done. This story would be better, if every time a backup finished, a DBA would be paged to physically change tapes, and file the old tape offsite, maybe a hundred and twenty miles away.

  • Tamm (unregistered)

    As a person who backs up once every decade or so I wouldn't think the process is big enough, even with the added paper copy. . . . thankfully, I am not that type of person. lmao

  • sobachatina (unregistered)

    Pardon my ignorance but I'm a programmer (who doesn't write database code) not a DBA.

    What is the significance of the 60 queries? Does it just emphasize the overkill of the solution because there are so few changes?

  • Leo (unregistered)

    And what if the power goes down citywide ? Katrina-like ? Tnen obviously the off-site backup can't be on the same state. I suggest they set up the first ever Moon backup facility. They'd be covered even in the case of Earth's complete obliteration by an asteroid!

  • PleegWat (unregistered)

    60 isn't much, even if that's only statements that actually modify the data.

  • (cs) in reply to Leo
    Leo:
    And what if the power goes down citywide ? Katrina-like ? Tnen obviously the off-site backup can't be on the same state. I suggest they set up the first ever Moon backup facility. They'd be covered even in the case of Earth's complete obliteration by an asteroid!
    But what if the moon crashes into the Earth? What if the sun goes nova? You have to be covered, man!
  • (cs) in reply to PleegWat
    PleegWat:
    60 isn't much, even if that's only statements that actually modify the data.

    Yes but those 60 pieces of data are VERY important. They control exactly how much poison/antidote to dispense every hour into all corporate coffee systems.

  • James Schend (unregistered) in reply to sobachatina

    Unless they have the craziest schema on earth (possible!), a database commit is never going to be more than maybe 50-100k of data, and probably much less. To add to that, an average database will have many queries that only select and don't modify or commit anything (and therefore don't change back-up-able data.)

    Being generous, they might be adding 1 MB an hour to this DB, absolute max. It's probably closer to 1 MB a day.

  • (cs) in reply to sobachatina

    60 Queries does not imply changes, that implies how many times the data is read for some purpose (application display, reporting, etc.). Usually queries >> changes > inserts > deletions.

    And yes, 60 queries an hour is ridiculously low for a system with this level of backup.

  • (cs)

    They need another backup server. In space. In case some aliens blow up the planet to make room for an interstellar highway.

  • some mailer (unregistered) in reply to Ted :(
    Ted :(:
    actually printed. The printout takes 5 reams of paper (2500 sheets), the better part of an hour to complete and kills a small forest in the process.

    You have to report that to GreenPeace. Nobody ever care to back-up forests

  • Ted :( (unregistered) in reply to evanm
    evanm:
    60 Queries does not imply changes, that implies how many times the data is read for some purpose (application display, reporting, etc.). Usually queries >> changes > inserts > deletions.

    And yes, 60 queries an hour is ridiculously low for a system with this level of backup.

    'Ted' again...

    When I originally sumbitted this WTF, the word "queries" was actually "transactions" to imply that more was happening than simple "SELECT's". We have about 2-600 hits to the DB/hour (depending on time of day, etc.) with about 10% of those hits resulting in some change in db data.

    As for our actual daily delta, it's typically <1Mb; so to put things in perspective, we were backing up 48Gb of data every night to capture a 1Mb change.

    BTW, great idea on the extra-solar back-up solution! I'll propose it to the Board at their next meeting... never know when our Sun's going to go Red-Giant ;)

  • chran (unregistered) in reply to PSWorx

    No, they need to build a large radio-telescope, like Arecibo.

    Then, every night they send out their backup! Restoring is a piece of cake! You just send out a space ship to the front of the data, and then you ...

    Well, if you ever need to restore, FIRST you need to invent a space ship that travels faster than light!

  • (cs) in reply to Ted :(

    They should have a fully-trained team of people sitting in a nuclear bunker. You never know when Ted and the rest of the staff can happen to be traveling on the same tramway, which coincidentally gets run over by a huge commercial airplane, and they all die. You have to back up your critical employees, people!

  • (cs) in reply to jkohen
    jkohen:
    They should have a fully-trained team of people sitting in a nuclear bunker. You never know when Ted and the rest of the staff can happen to be traveling on the same tramway, which coincidentally gets run over by a huge commercial airplane, and they all die. You have to back up your critical employees, people!

    So the fully trained team in the bunker would actually be clones of Ted and his staff?

  • Little Green Men, Inc. (unregistered) in reply to chran

    <quote>No, they need to build a large radio-telescope, like Arecibo.

    Then, every night they send out their backup! Restoring is a piece of cake! You just send out a space ship to the front of the data, and then you ...

    Well, if you ever need to restore, FIRST you need to invent a space ship that travels faster than light!</quote>

    Hello from Oometek, Inc. The ultimate in outsourcing.

    Are you storing your data in only one solar system? We can store your back-ups in up to 10 different partner solar systems.

    We at Alpha Centauri will warehouse your data for you for a modest fee. The technologies we have available at facilities are light-years ahead of any other out-sourcing organization. And we can offer Earth the shortest data retrieval turn-around time of any company in the galaxy.

    Ask about creating an oort-cloud facility for express data delivery!

    Regards,

    Plik-nik

  • (cs) in reply to Look at me! I'm on the internets!
    Look at me! I'm on the internets!:
    I would have had IT staff do a printout of the entire database every night and fax it to a branch office where it could be converted to microfilm.

    ...then the microfilm would be set on a wooden table, have its picture taken which will then be scanned. OCR would then...

  • Adam (unregistered)

    I worked at a company where the offsite backups were kept in a salt mine in a mountain, there was a company that managed storage there. At a disaster recover meeting, one team member was concerned that in the event of an earthquake we wouldn't be able to retrieve the backup tapes because all of the helicopters would be used for rescue work. We told him not to worry, none of us would be there to restore the tapes in that case.

  • mh (unregistered) in reply to Look at me! I'm on the internets!
    Look at me! I'm on the internets!:
    I would have had IT staff do a printout of the entire database every night and fax it to a branch office where it could be converted to microfilm.
    Cool! OCR as a viable restore strategy, anyone?
  • (cs) in reply to Look at me! I'm on the internets!
    Look at me! I'm on the internets!:
    I would have had IT staff do a printout of the entire database every night and fax it to a branch office where it could be converted to microfilm.

    Obligatory mention "if photographed on a wooden table".

  • (cs)

    60 per hour! That's almost eighty an hour!

  • (cs) in reply to sobachatina
    sobachatina:
    What is the significance of the 60 queries? Does it just emphasize the overkill of the solution because there are so few changes?
    Sort of.

    The real WTF here is slightly hidden (Ted's posting makes it clearer). The 4 replication servers with a tape backup each is not a WFT. Slighty overkill maybe, but I'd rather have that than the far more common case of "Backup? What's that?". Having those 4 servers in the same room adds a nice touch of "defeating the purpose", but the real WTF is that instead of the tiny delta that such a small transaction volume generates, they do MANY multiple full backups.

    Of course, the only reason that that's possible at all IS the small transaction volume, otherwise they'd run into nearly unsurmountable synchronization problems.

  • (cs)

    gosh. this article was great. it reminded me to do my backup. I'd like to request we do a backup article on a semi-random basis at least once every other alternating thursday after a tuesday with a full moon with respect to the antarctic ice station on the leeward side of the northernmost point of the iceberg that just won't stay in the same damn place.... ^%$@&% global warming!

    whoops --- time to leave, i'll just do the backup tomorrow.

  • Peter (unregistered) in reply to mh
    mh:
    Look at me! I'm on the internets!:
    I would have had IT staff do a printout of the entire database every night and fax it to a branch office where it could be converted to microfilm.
    Cool! OCR as a viable restore strategy, anyone?

    With, of course, obligatory correction processes unless OCR has advanced FAR beyond where it was a couple of years ago. Ugh. (I was thinking the same thing, though.)

  • Franz Kafka (unregistered) in reply to Ted :(
    Ted :(:
    evanm:
    60 Queries does not imply changes, that implies how many times the data is read for some purpose (application display, reporting, etc.). Usually queries >> changes > inserts > deletions.

    And yes, 60 queries an hour is ridiculously low for a system with this level of backup.

    'Ted' again...

    When I originally sumbitted this WTF, the word "queries" was actually "transactions" to imply that more was happening than simple "SELECT's". We have about 2-600 hits to the DB/hour (depending on time of day, etc.) with about 10% of those hits resulting in some change in db data.

    As for our actual daily delta, it's typically <1Mb; so to put things in perspective, we were backing up 48Gb of data every night to capture a 1Mb change.

    BTW, great idea on the extra-solar back-up solution! I'll propose it to the Board at their next meeting... never know when our Sun's going to go Red-Giant ;)

    So, what you're saying is that I could run their entire DB on my desktop without really noticing.

  • Ted :( (unregistered) in reply to Franz Kafka
    Franz Kafka:
    So, what you're saying is that I could run their entire DB on my desktop without really noticing.
    I'm saying that you could probably run it on your cell phone, while making a call and synchronizing your contact list without the phone so much as blinking. I'm sure running it on your desktop would be overkill... :þ
  • Top Cod3r (unregistered)

    The real WTF is Ted doesn't seem to understand how backups work. At any one time I keep a full source code backup in a separate filesystem partiton hourly (my clients don't pay for me to re-do work, you see). I also have a keychain with 8 usb drives attached to "backup the backup" one for each hour of the day I work, except lunch time.

    Ted will learn the hard way once he is fired after his backup failed. And Beaker will be able to say "I told you so". Maybe Ted will then become a guy who keeps 16 backups.

  • Darien H (unregistered) in reply to Little Green Men, Inc.
    Little Green Men:
    <quote> Ask about creating an oort-cloud facility for express data delivery! Regards, Plik-nik

    My company has also had great experiences dealing with another local milky-way concern. You can reach their trade master by hyperwave:

    [email protected]

  • Josh (unregistered) in reply to Ted :(

    You could try that, but the weight of all the servers would probably crush your desk.

  • mabinogi (unregistered) in reply to mh
    mh:
    Look at me! I'm on the internets!:
    I would have had IT staff do a printout of the entire database every night and fax it to a branch office where it could be converted to microfilm.
    Cool! OCR as a viable restore strategy, anyone?
    Actually, Microfilm is not such a bad idea - it lasts a hell of a lot longer than optical disk or magnetic media.
  • Beaker's little buddy (unregistered)

    Ooh, I think I used to work with this "Beaker" fellow.

  • Bubba (unregistered)

    So the database is covered. Have you spoken with mgmt about the resumption of business activities after a disaster (tidalwave, tornado, hurricane, etc) so that even if you get the data back online quickly, the business will actually function again? Time to make the rest of the folks scramble for a while... :)

    captcha = digdug

  • Rick (unregistered) in reply to Top Cod3r
    Top Cod3r:
    The real WTF is Ted doesn't seem to understand how backups work. At any one time I keep a full source code backup in a separate filesystem partiton hourly (my clients don't pay for me to re-do work, you see). I also have a keychain with 8 usb drives attached to "backup the backup" one for each hour of the day I work, except lunch time.

    CAPTCHA: ewww

    kinda says it all, really.

  • Anon (unregistered) in reply to Top Cod3r
    Top Cod3r:
    Ted will learn the hard way once he is fired after his backup failed. And Beaker will be able to say "I told you so". Maybe Ted will then become a guy who keeps 16 backups.
    Why! Why! Why! Why didn't they use six thousand and one backups?! When will the fools learn?!

    What happens "if" all 16 backups are corrupted? What happens if backup #11 has corruption that isn't detected? There are genuine mission critical systems (meaning "lives will be lost if it fails" and not just "someone could have to retype a whole day's worth of development") with saner backup strategies than Beaker's badly mismanaged 16 backup system.

    <sarcasm_detector enabled="duh">Honestly, some people understand backups so badly that they make hourly backups, instead of having each and every keypress added to a backup log that gets copied to 283,476 USB drives - thus risking losing a whole hour of work that they won't be paid to redo. Honestly, how could anyone get by just hoping that none of their backups will be corrupted? If you have N backups, any fool knows you need N+1 incase the first N are corrupted.</sarcasm_detector>

  • Old Wolf (unregistered)

    Pfft, who needs backups. I've never backed up my data and I've never had a pr+++

    NO CARRIER

  • DylanW (unregistered) in reply to KattMan
    KattMan:
    Yes but those 60 pieces of data are VERY important. They control exactly how much poison/antidote to dispense every hour into all corporate coffee systems.
    I would love to be a part of the meeting where they explain this.

    "The backups are critical because this database contains the formula for the antidote." "The antidote to what?" "To the poison you just drank!"

    Captcha: yummy

  • Marc (unregistered) in reply to Darien H
    Darien H:

    How much does it cost to find out why your bridge just turned purple?

  • Elena (unregistered)

    Pardon my ignorance, but how long would it take for a change to be backed-up for the first time at this rate(60/hour)?

  • (cs) in reply to Top Cod3r
    Top Cod3r:
    The real WTF is Ted doesn't seem to understand how backups work. At any one time I keep a full source code backup in a separate filesystem partiton hourly (my clients don't pay for me to re-do work, you see). I also have a keychain with 8 usb drives attached to "backup the backup" one for each hour of the day I work, except lunch time.

    Ted will learn the hard way once he is fired after his backup failed. And Beaker will be able to say "I told you so". Maybe Ted will then become a guy who keeps 16 backups.

    Shush. And get back under your bridge, troll.

Leave a comment on “A Backup's Backup's Backup”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article