• Nobby (unregistered)

    '10MB database, 100MB of disk space and 100GB of bandwidth'

    Soon fill the 10 meg database up with that bandwidth bein used.

  • Goatboy (unregistered)

    What a sad story :(

  • Ouch! (unregistered)

    Isn't there a law requiring a company to have backups? If that story hasn't been exaggerated out of proportion, really, it's a miracle they lasted more than three months.

  • (cs)
    the company was upset that its Web site ran incredibly slow. {...} it would still take upward of a couple seconds for a page to load.
    Oh, I weep for them, I really do.

    Seriously, what kind of speed did they think they deserved, choosing the $75/month package?

  • (cs) in reply to Ouch!
    Ouch!:
    Isn't there a law requiring a company to have backups? If that story hasn't been exaggerated out of proportion, really, it's a miracle they lasted more than three months.

    and you know how many companies ignore regulations?

  • (cs)

    I've never run an ISP, but I do host domains, and I certainly make backups of all the files. I would expect a real ISP to do the same, but apparently that is not the case here.

  • Anon (unregistered)
    To say the developer panicked is akin to describing a quadruple amputation as a mere flesh wound.

    BLACK KNIGHT: Oh? All right, we'll call it a draw. ARTHUR: Come, Patsy. BLACK KNIGHT: Oh. Oh, I see. Running away, eh? You yellow bastards! Come back here and take what's coming to you. I'll bite your legs off!

  • (cs)

    For a second, I thought this might be about Pets.com, but then I saw that they had stores and actually made money.

  • synp (unregistered) in reply to Ouch!
    Ouch!:
    Isn't there a law requiring a company to have backups?

    Only in well regulated industries - health care, defense contractors, financial industry.

  • Tweet (unregistered) in reply to Ouch!
    Ouch!:
    Isn't there a law requiring a company to have backups?
    There's not even a law requiring a store to have computers. It's just good business sense.

    Not having backups could be understood, I suppose, if you never ever in your entire life saw any kind of machine ever malfunction or quit working in any way. Nor did anyone on your crew. And you thought you could put your entire business on a machine far more complicated than a helicopter without anyone ever getting any kind of training, or hiring someone with even a month of computer experience. And all the other managers in the company were as uninformed and thoughtless as you.

    In short, this company deserved to fail -- no, needed to fail. And the sooner the better. With no bailout option. The only downside is the innocent employees.

  • synp (unregistered) in reply to dpm
    dpm:
    I've never run an ISP, but I do host domains, and I certainly make backups of all the files. I would expect a real ISP to do the same, but apparently that is not the case here.

    Mmm, maybe the $75/mo package doesn't include backups.

    OTOH, what if it was your disk drive crashing that killed their website, or worse, their company? I guess you would want to have backups for that, no?

  • wbo (unregistered) in reply to dpm
    dpm:
    I've never run an ISP, but I do host domains, and I certainly make backups of all the files. I would expect a real ISP to do the same, but apparently that is not the case here.

    Think of the privacy! I wouldn't want my ISP to have a backup of my porn^H^H^H^Hfiles!

    I would expect a content hoster to have a backup, but not for this cheap package.

    And I would like to add that in my opinion this is a Genuine WTF©.

  • St Mary's Hospital for the Non-Euclidean Geography (unregistered)

    Hmmm, where is the Tri-State Area?

    I thought of Hawaii. But swimming in any direction, you run ashore in either California, Washington and Oregon. And Alaska.

    Does anybody have a clue?

  • (cs)
    was forced to close every one of its stores -- laying off several hundred people along the way.

    Yeah, but what happened to all the animals?

  • Yeah (unregistered)

    This is why you "start transaction" before deleting 10 million rows.

  • Steve the Cynic (unregistered) in reply to SlyEcho
    SlyEcho:
    was forced to close every one of its stores -- laying off several hundred people along the way.

    Yeah, but what happened to all the animals?

    Barbecue?

  • JoPoser (unregistered)

    Nitpicking....

    They were an IAP that also hosted. an ISP does host by default.

    What's an IAP? AO-effin-L. Internet Access Provider; I'm an ISP but not an IAP. Can't tell you how many people think that ISP means getting online and are upset when they talk to me on the phone.

  • some guy (unregistered) in reply to SlyEcho
    SlyEcho:
    was forced to close every one of its stores -- laying off several hundred people along the way.

    Yeah, but what happened to all the animals?

    burp

  • JoPoser (unregistered) in reply to St Mary's Hospital for the Non-Euclidean Geography

    Chicago.

    Illinois, Indiana and Wisconsin.

  • Jim Lard (unregistered) in reply to dpm
    dpm:
    I've never run an ISP, but I do host domains, and I certainly make backups of all the files. I would expect a real ISP to do the same, but apparently that is not the case here.

    Most ISPs I've used make it very clear that they don't back up customer data by default, although many do offer it as an additional service. At the end of the day, if you're a business, your data is being hosted at a third-party server and you want it backed up, it's important to get a solid contract detailing what's to be backed up, when, how, the types of restore available, etc etc etc.

  • Anonymous (unregistered) in reply to dpm
    dpm:
    I've never run an ISP, but I do host domains, and I certainly make backups of all the files. I would expect a real ISP to do the same, but apparently that is not the case here.
    For $75/month? Not a chance! That was a fundamental part of the WTF, the fact that a major company was running their entire operation on the cheapo $75/month hosting service. You get what you pay for in this world and clearly they paid for the barest minimum they could. Unfortuntaley for them, that didn't include a backup service.
  • Bob (unregistered) in reply to St Mary's Hospital for the Non-Euclidean Geography

    The tri-state area could be a couple of places. Once possibility is NY/NJ/Eastern PA. Another is Ohio/WV/Western PA. I've even heard PA/DE/MD referred to as the tri-state area.

  • Anon (unregistered) in reply to JoPoser
    JoPoser:
    Chicago.

    Illinois, Indiana and Wisconsin.

    Or Cincinnati - Ohio, Kentucky, Indiana. And a couple of years back a chain of local pet shops did go bankrupt (can't remember the name though).

  • (cs) in reply to Ouch!
    Ouch!:
    Isn't there a law requiring a company to have backups?

    Sort of. SOx (Sarbanes Oxley Act) requires publicly traded companies to maintain many kinds of records (not necessarily backups) for several years. If they were privately held....

  • (cs) in reply to dpm
    dpm:
    I've never run an ISP, but I do host domains, and I certainly make backups of all the files. I would expect a real ISP to do the same, but apparently that is not the case here.

    I would expect a real business to backup its own data, apparently that wasn't the case here either. Along with a few other cases. Like not hosting your company's entire IT infrastructure on a remote server.

  • Anonymous (unregistered)

    It's pretty much impossible to feel for these jokers. Running your entire business through your website is bad enough; running your entire business through your website which is powered by a PHP content management framework is even worse; but doing all this and then hosting it on a bargain-basement hosting deal and not bothering to make backups? They absolutely deserved to go under and good riddance to another technically clueless company.

  • (cs) in reply to Bob
    Bob:
    The tri-state area could be a couple of places. Once possibility is NY/NJ/Eastern PA. Another is Ohio/WV/Western PA. I've even heard PA/DE/MD referred to as the tri-state area.

    Also "DelMarVa" (Delaware/Maryland/Virginia), since they share a peninsula.

  • (cs) in reply to St Mary's Hospital for the Non-Euclidean Geography
    St Mary's Hospital for the Non-Euclidean Geography:
    Hmmm, where is the Tri-State Area?

    I thought of Hawaii. But swimming in any direction, you run ashore in either California, Washington and Oregon. And Alaska.

    Does anybody have a clue?

    Solid/Liquid/Gas

  • blah (unregistered)

    You idiots! The tri-state area is obviously enum { True, False, FileNotFound }!

  • Neil (unregistered) in reply to Smash King
    Smash King:
    St Mary's Hospital for the Non-Euclidean Geography:
    Hmmm, where is the Tri-State Area?

    I thought of Hawaii. But swimming in any direction, you run ashore in either California, Washington and Oregon. And Alaska.

    Does anybody have a clue?

    Solid/Liquid/Gas
    Plasma?

  • cfreak (unregistered)

    Wow ... I'm going to have to call BS on this story.

    First of all ... not knowing how much money that had? WTF? One would assume they have bank-accounts. That typically gives one a good idea. Its not like their bank was storing records there. That could have given them plenty of information on the amount of money likely lost and a basis for a lawsuit.

    Their vendors probably would have been happy to reprint invoices if some were unpaid and missing.

    The biggest problem would be is if they had unpaid invoices ... but if they were a retail chain its not all that likely they would have had many.

    TRWTF is that the ISP had no backups. They deserved to be sued. It isn't their place to determine if a customer's database is crappy. Even if the $75 account meant no backups were made, the developer should have made backups before any work was done!

    I suspect that if this story is true at all there's a lot more to it than what is here

  • eliac (unregistered)

    If the DB "grew into a vast monolith over seven years", how was it a surprise that they needed more than a 10MB database? Also, am I the only one feeling slightly uncomfortable at the thought of an ISP representative so closely inspecting the data in a customer's DB?

  • (cs) in reply to dpm

    Remember the ISP offers ala carte services. With how paranoid this company seemed to be they may not have trusted the ISP to do their backups and may have even opted out thinking they were trying to be sold something unnessasary.

  • bosox1256 (unregistered) in reply to Bob

    I think the actual Tri-State area is NY/NJ/Fairfield County, CT.

  • BK (unregistered) in reply to Anonymous

    I agree that they deserve what they got.

    Still I cannot help but wonder how this could have possibly happened. One clue is in "The owners of MegaPetCo, however, suspected ..." - did the owners make technical decisions there, while remaining totally unaware of the potential consequences?

  • StDoodle (unregistered)

    The really sad part is that, for what they paid in a month or two's overages, they could have hired someone to clean up their code & database enough to limp along fairly well on even their low-tier plan. They wouldn't have even needed a seasoned developer; they could have found someone at the local community college who would have at least been able to see the need for a few more tables in the database, if nothing else.

    Sadly, I've heard a few stories like this from my wife, who's worked a lot of retail. I've commented to her that the software & network setup sounds horrible at several places she's worked over the years, and she just shrugs it off as if it's expected.

  • Your father (unregistered)

    Just a small thought about an important (but silent) actor in this story : the guy who "unluckily" fired the delete query.

    honnestly, let along the immediate consequences, don't you think he must have a little "pride" over all this ? ^^

  • Bob (unregistered) in reply to kmarsh
    kmarsh:
    Bob:
    The tri-state area could be a couple of places. Once possibility is NY/NJ/Eastern PA. Another is Ohio/WV/Western PA. I've even heard PA/DE/MD referred to as the tri-state area.

    Also "DelMarVa" (Delaware/Maryland/Virginia), since they share a peninsula.

    Who knows, growing up NY/NJ/E. PA was the tri-state area, and I was in my own little world, so it didn't dawn on me until I moved across the state that it could refer to more than one area.

  • Ickithus (unregistered)

    Ok, I'll add one too, Evansville Indiana. Indiana, Kentucky, Illinois. Seriously, there are a lot more tri-state areas than you'd think.

  • douglas (unregistered) in reply to cfreak
    cfreak:
    TRWTF is that the ISP had no backups. They deserved to be sued. It isn't their place to determine if a customer's database is crappy. Even if the $75 account meant no backups were made, the developer should have made backups before any work was done!
    Uh, the developer in question was working for MegaPetCo, not the ISP. All anyone at the ISP did was take a look and shake their heads without changing anything. So, yes the developer should have made a backup, but that has nothing to do with whether the ISP was at fault.
  • p2kmon (unregistered)

    I believe the story. I worked at a Microsoft for awhile and we had a VERY, VERY large telecom call up and they lost some major harddrives. They had Backups for 6 years.

    Problem is that NO one EVER tested the backups. So even if a backup is done and you don't test them, you could be in a whole lot of problems. Not a single backup worked.

  • sep332 (unregistered) in reply to Neil
    Neil:
    Smash King:
    St Mary's Hospital for the Non-Euclidean Geography:
    Hmmm, where is the Tri-State Area?

    I thought of Hawaii. But swimming in any direction, you run ashore in either California, Washington and Oregon. And Alaska.

    Does anybody have a clue?

    Solid/Liquid/Gas
    Plasma?
    Bose-Einstein condensate?

  • EdB (unregistered) in reply to St Mary's Hospital for the Non-Euclidean Geography

    New York New Jersey Pennsylvania.

  • D (unregistered)

    MegaPetCo? Are you sure it wasn't MegaPetSmart?

  • blindman (unregistered)

    10mb for a database? You must be joking. I wouldn't store my grocery list in a 10mb database.

  • (cs)

    The code monkey survived that? They didn't hang him upside down from the ceiling?

  • Ben4jammin (unregistered) in reply to p2kmon
    p2kmon:
    I believe the story. I worked at a Microsoft for awhile and we had a VERY, VERY large telecom call up and they lost some major harddrives. They had Backups for 6 years.

    Problem is that NO one EVER tested the backups. So even if a backup is done and you don't test them, you could be in a whole lot of problems. Not a single backup worked.

    yea, stories like this are why it is written into my job description that we have monthly test restores (we do backup to disk and tape). And I have to document what was restored when and whether it was successful. It is the restore part that actually makes it a backup...otherwise its just a tape that you can use as a paperweight as you update your resume...

  • josephdietrich (unregistered)

    Holy moly. That's not death by delete. That's death by stupidity, with an assist by delete.

  • OldHand (unregistered) in reply to Yeah
    Yeah:
    This is why you "start transaction" before deleting 10 million rows.

    BEGIN; DELETE ... FROM ...; COMMIT;

    There. Put it all on one line to make it compact, no wasteful newlines.

  • (cs)

    Although I agree that he should have put the deletes in a transaction, its like most other wtf's here. The production data is the test data. Lets try just writing up this quick delete query... oops..

Leave a comment on “Death by Delete”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article