• Zapp Brannigan (unregistered) in reply to Yeah
    Yeah:
    This is why you "start transaction" before deleting 10 million rows.
    Put it in your environment settings. Other than nuking from space, it's the only way to be sure.
  • JdFalcon04 (unregistered) in reply to Ickithus
    Ickithus:
    Ok, I'll add one too, Evansville Indiana. Indiana, Kentucky, Illinois. Seriously, there are a lot more tri-state areas than you'd think.

    I'll throw one in as well... New Hampshire, Mass, Vermont. They have (had? it's been a while) a "tri-state lottery."

  • BK (unregistered) in reply to cfreak
    cfreak:
    TRWTF is that the ISP had no backups. They deserved to be sued. It isn't their place to determine if a customer's database is crappy. Even if the $75 account meant no backups were made, the developer should have made backups before any work was done!

    I think that when the ISP presented alternative plans at the beginning of the whole story, they must have also explained in no uncertain terms that they do NOT backup user databases (anyway, not in the cheapest plan the customer selected). That would explain why shop witdrew action - although there might have been many reasons.

    No ISP would backup gigabytes of data without payment, because maintaining these backups costs real money (offsite copy? tape retention? monitoring and managing backups? etc.). I know because it was part of my job, some 10 years ago (different ISP, though). Before anyone jumps to "the database was only few gigabytes", please consider that it was not the only customer on the cheapest plan. Quite likely there would have been lots and lots of data owned by customers on this plan.

    Captcha : I saluto thee

  • SR (unregistered) in reply to josephdietrich
    josephdietrich:
    Holy moly. That's not death by delete. That's death by stupidity, with an assist by delete.

    Deletes don't kill businesses, crap developers do.

  • Dennis (unregistered) in reply to JdFalcon04

    Don't forget ArkLaTex!

  • nico (unregistered)

    Well, I have my website on a 35 euro/year domain and it does have DB backup so I don't see why a $75/month service shouldn't

  • (cs) in reply to pitchingchris
    pitchingchris:
    Although I agree that he should have put the deletes in a transaction, its like most other wtf's here. The production data is the test data. Lets try just writing up this quick delete query... oops..

    It's a php website. So we're talking MySQL databases. The whole database is a single table, and MySQL still defaults to MyISAM instead of INNODB, so you probably need the sort of skill that you'd normally use to normalize a database to even know about transactions and that not all mysql tables are transaction capable by default.

    So, in conclusion, what transaction safe database engine do you think they're using?

    The fall of the company had nothing to do with its database, and everything to do with its crappy management.

  • blindio (unregistered)

    here, and I thought Tri-State area was a Visual Basic boolean.

  • Junkie (unregistered)

    "The database was incredibly simple: a single table with hundreds of columns."

    Noone has a problem with this?

    Captcha: populus - man, I used to love that game!

  • (cs) in reply to JoPoser
    JoPoser:
    Nitpicking....

    They were an IAP that also hosted. an ISP does host by default.

    What's an IAP? AO-effin-L. Internet Access Provider; I'm an ISP but not an IAP. Can't tell you how many people think that ISP means getting online and are upset when they talk to me on the phone.

    I'm not sure if you just failed to communicate properly, so I'll verify. Are you indicating that you are an Internet Service provider, who does not provide internet service? If so, you are surprised and/or dismayed when potential customers call inquiring about internet service only to find out that you, as an internet service provider do not, in fact provide internet service?

  • (cs) in reply to Dennis
    Dennis:
    Don't forget ArkLaTex!

    Just for the record, Oregon, Washington and Idaho is NOT called the tri-state area..... you know, just in case you wanted to know another useless piece of trivia.

  • (cs) in reply to sep332
    sep332:
    Bose-Einstein condensate?
    This case sounds a bit more like a Bozo-Einstein condensate. Except without the Einstein.
  • (cs)

    I think this is a perfect example of "technical debt" run amok.

    When the company started (well, before it started, from the sound of it), they had a well functioning (but unscalable) website and database. Once their business grew by leaps and bounds, management made the hideously costly mistake of NOT upgrading their infrastructure and instead patching on new columns, etc., as needed.

    The IT supervisor there was probably either A) clueless or B) screaming his head off at management for the entire life of the company.

  • Harrow (unregistered)

    That's what you get for trying to use cloud computing before it's a mature technology.

    -Harrow.

  • BBT (unregistered)

    http://lmgtfy.com/?q=tri-state+area

  • SR (unregistered) in reply to Junkie
    Junkie:
    "The database was incredibly simple: a single table with hundreds of columns."

    Noone has a problem with this?

    Captcha: populus - man, I used to love that game!

    It's simple. Lunacy but simple.

  • MFD Lover (unregistered) in reply to Ouch!
    Ouch!:
    If that story hasn't been exaggerated out of proportion

    You are new here, right?

  • idlewire (unregistered) in reply to SlyEcho

    They were unfortunately in the database as well.

  • d.k. Allen (unregistered)

    If I were providing contract services, backups would be a billable option, not an automatic given. If I committed to backing up their data, and a restore failed for any reason, my customer is going to sue me to death. I'm for sure gonna charge extra for that risk. OTOH, All data would be RAID'd, certainly. Not a backup plan, but protection against failing drives.

    Tri-state area: For Orlando, we have Vermont, New Mexico, and Alaska.

    Captcha: commoveo ? What the f... oh, wait...

  • Bruce W (unregistered) in reply to Ickithus
    Ickithus:
    Ok, I'll add one too, Evansville Indiana. Indiana, Kentucky, Illinois. Seriously, there are a lot more tri-state areas than you'd think.

    I was thinking of the Four Corners area. Oh wait, that's the QUAD-state area.

  • GT (unregistered) in reply to BBT

    Thank you. There is a cute little "tri-state gas station" in Port Jervis where NY, PA, and NJ meet, but come on people, I can't believe anyone would think of that.

    Unless you are talking about NY/NJ/CT, you have to qualify your "tri-state area".

  • aaawww (unregistered)

    benelux?

  • WW (unregistered) in reply to dpm
    dpm:
    I've never run an ISP, but I do host domains, and I certainly make backups of all the files. I would expect a real ISP to do the same, but apparently that is not the case here.
    When you rent a virtual dedicated server (or a real dedicated server) the backups are generally your responsibility, not the ISP's. Given that this ISP seems to have sold a lot of services a la carte, the company probably decided not to pay for the backup service, and then also didn't do it themselves.
  • (cs) in reply to Ouch!
    Ouch!:
    Isn't there a law requiring a company to have backups? If that story hasn't been exaggerated out of proportion, really, it's a miracle they lasted more than three months.
    Maybe they kept their backups in more columns of the database...
  • Zygo (unregistered) in reply to Anonymous
    Anonymous:
    dpm:
    I've never run an ISP, but I do host domains, and I certainly make backups of all the files. I would expect a real ISP to do the same, but apparently that is not the case here.
    For $75/month? Not a chance! That was a fundamental part of the WTF, the fact that a major company was running their entire operation on the cheapo $75/month hosting service. You get what you pay for in this world and clearly they paid for the barest minimum they could. Unfortuntaley for them, that didn't include a backup service.

    $4,281.32/month is not the barest minimum they could have paid.

    Their problem was that they ignored people smarter than them advising them repeatedly that they were paying punitive prices for being too stupid to pick the right service package.

  • Leo (unregistered) in reply to GT
    GT:
    Thank you. There is a cute little "tri-state gas station" in Port Jervis where NY, PA, and NJ meet, but come on people, I can't believe anyone would think of that.

    Unless you are talking about NY/NJ/CT, you have to qualify your "tri-state area".

    Why would you not have to qualify it then? Oh, right, because NYCers think they're the center of the universe. Sorry, but most people think it's a hellhole.

    And you always have to qualify your tri-state area unless you're using it in an obviously local context to other local people.

  • Zygo (unregistered) in reply to Junkie
    Junkie:
    "The database was incredibly simple: a single table with hundreds of columns."

    Noone has a problem with this?

    Nah, that's normal 'round here.

    It would be more notable if the database was actually well designed--that would indicate that at least one person in MegaPetCo's employ had a technical clue, which would make the total loss of all business operations data at the end of the story even more poignant.

  • notme (unregistered)

    Okay, so I guess, for some weird reason, hosting is more expensive in the US than in europe, but still - how come 75 Dollars a month bought the man only a measly little webspace on a shared host and not an entire dedicated server?

    Not that it would have saved him, but still, those prices just strike me as way out there. Maybe Alex forgot to mention that this story is more than 10 years old?

  • (cs) in reply to Junkie
    Junkie:
    "The database was incredibly simple: a single table with hundreds of columns."

    Noone has a problem with this?

    Seen it before, both in my work and on here.

  • SNF (unregistered) in reply to Junkie
    Junkie:
    "The database was incredibly simple: a single table with hundreds of columns."

    Noone has a problem with this?

    Of course I have a problem with it, but a) It's been seen dozens of times before b) The remaining wtfery far, far, outshines it

  • Gio (unregistered) in reply to St Mary's Hospital for the Non-Euclidean Geography

    I always thought tri-state meant NY, NJ and CT: http://en.wikipedia.org/wiki/Tri-State_Region

  • someguy (unregistered)

    what morons, if you've got a customer who's shelling-out an extra $4,000/mo in fees despite cheaper alternatives which you have offered, cling to that customer for dear life! Upgrade his plan behind the scenes, back up his files regularly, hire someone to keep an eye on their site full-time and /still/ wind up $2000 better for it. If you couldn't keep their site up despite having $4000 in free money every month, you really have only yourselves to blame.

    Seriously, what an absolutely worthless company.

  • Ethan Qix (unregistered) in reply to SlyEcho
    SlyEcho:
    Yeah, but what happened to all the animals?

    No idea. burp

  • similis (unregistered) in reply to Junkie
    Junkie:
    "The database was incredibly simple: a single table with hundreds of columns."

    Noone has a problem with this?

    Captcha: populus - man, I used to love that game!

    Nah, that is all too common around here. ;)

    I sortof went "Oh, they didnt fuck up their DB design?" when I read the first few words there, then reality hit me like a freightrain hauling depeted uranium by the wagonfulls. ;)

    That is, this is TDWTF and its kindof like "YE who enter here, abandon all home/reason" ;)

  • SR (unregistered) in reply to SNF
    SNF:
    wtfery

    Superb!

  • Americium (unregistered) in reply to Satanicpuppy
    Satanicpuppy:
    It's a php website. So we're talking MySQL databases. The whole database is a single table, and MySQL still defaults to MyISAM instead of INNODB, so you probably need the sort of skill that you'd normally use to normalize a database to even know about transactions and that not all mysql tables are transaction capable by default.

    So, in conclusion, what transaction safe database engine do you think they're using?

    The fall of the company had nothing to do with its database, and everything to do with its crappy management.

    Transaction management still has its limits. The database server has to track all the changes in some buffer. In Oracle, this is referred to as the ROLLBACK SEGMENT.

    DELETE FROM THE_TABLE; requires a ROLLBACK SEGMENT holding all the rows. A GB database is too large to handle, period. The transaction fails due to memory or disk space limits.

    The poor developer couldn't rollback a transaction!

  • (cs) in reply to similis
    similis:
    That is, this is TDWTF and its kindof like "YE who enter here, abandon all home/reason" ;)
    So if we read TDWTF we'll end up homeless? 'scuse me, mister, can you help me out? Just a little spare change, something to help me get a bottle^H^H^H^H^H^Hbite to eat?
  • FragFrog (unregistered) in reply to eliac
    eliac:
    Also, am I the only one feeling slightly uncomfortable at the thought of an ISP representative so closely inspecting the data in a customer's DB?
    On what is technically a 'shared hosting' account, no. Most shared hosting accounts are used by people clueless as to what they do online, which invariably leads to either severe restrictions in what the customer can do, or the occasional spambot in one of your clients account.

    To this end most ISP's will have a clause in their contract allowing them to inspect your account, especially if there is good cause for it. Laws vary between nations, but generally the ISP can even be held responsible for the content it hosts for you (which is why take-down notices against the Pirate Bay for instance are also directed at its hosting company).

  • Bram (unregistered) in reply to dpm

    Backup? On a $75/mo package? Mwahahahaha.... Sorry, Dogbert moment.

    The usual ISP deal at that price level is that they provide space for you to make your files available to the internet, you're responsible for having backups &c. At additional cost they may be able to offer you a backup solution as well. The fully hosted, supported, backed up applications providers tend to be "a bit" more expensive. Which isn't to say that at $4K/month they couldn't have come close to that level of service if they'd switched over to a plan/provider more appropriate to their requirements.

  • Zygo (unregistered) in reply to cfreak
    cfreak:
    Wow ... I'm going to have to call BS on this story.

    First of all ... not knowing how much money that had? WTF? One would assume they have bank-accounts. That typically gives one a good idea. Its not like their bank was storing records there. That could have given them plenty of information on the amount of money likely lost and a basis for a lawsuit.

    Sure, I'll just look up the account numbers for you...oh, wait, I can't.

    Also, if you can find your money in bank accounts this way, you technically haven't lost it, so there's nothing to sue over. It's all the business records that drive processes that lead to the movement of money that they've really lost.

    cfreak:
    Their vendors probably would have been happy to reprint invoices if some were unpaid and missing.

    So, if I was a vendor, I could just give all the unpaid invoices to a collection agency, just because I have no sympathy for people who don't make even a minimal effort to make backups. Now the company has to convince the collection agency that they aren't owed exactly what I said they were owed, and that can be hard without any kind of business records.

    Even if their vendors were kind to them, the IRS won't be. The IRS wants records, otherwise there's nothing to audit. The accountant who prepares their tax returns will want records.

    cfreak:
    The biggest problem would be is if they had unpaid invoices ... but if they were a retail chain its not all that likely they would have had many.

    They'd be in trouble if they started missing payments to their creditors. Creditors talk to each other, and "my dog ate my IT infrastructure" is going to affect the company's credit rating. The company might lose their operating LoC's, or they may be faced with higher interest rates on future credit.

    They also have employee payroll. If the branch-level managers are smart, they've got printed records for all the payroll data so they can reenter it into the payroll system (after they rewrite the payroll system, because it was written in PHP and was deleted with the rest of the database). Of course, those managers were probably diligently entering the data into the database instead of keeping paper records...

    While all the employees are redoing inventory (which has been deleted), calculating daily sales numbers by hand (since the software that counts sales was in the database and has also been deleted), and trying to "remember" how many hours they worked since the last payroll run, the stores are hemorrhaging money due to over- and understocking, and any goodwill they have with their suppliers probably evaporated five paragraphs ago.

    Their competitors are completely unaffected by the fact that MegaPetCo is suddenly running a mature chain of stores with no IT infrastructure and untrained staff. The competitors will have no change in their operating overheads, their staff will be familiar with their IT systems, their shelves will be filled with the right amount of product, and their suppliers will have orders to ship new products to the stores at the right times to avoid running out without filling the store with unsold product.

  • William Furr (unregistered) in reply to sep332
    sep332:
    Neil:
    Smash King:
    St Mary's Hospital for the Non-Euclidean Geography:
    Hmmm, where is the Tri-State Area?

    I thought of Hawaii. But swimming in any direction, you run ashore in either California, Washington and Oregon. And Alaska.

    Does anybody have a clue?

    Solid/Liquid/Gas
    Plasma?
    Bose-Einstein condensate?

    Granularity!

    http://www.wired.com/wiredscience/tag/granularity/

  • (cs) in reply to JoPoser
    JoPoser:
    Nitpicking....

    They were an IAP that also hosted. an ISP does host by default.

    What's an IAP? AO-effin-L. Internet Access Provider; I'm an ISP but not an IAP. Can't tell you how many people think that ISP means getting online and are upset when they talk to me on the phone.

    "An Internet service provider (ISP, also called Internet access provider, or IAP) is a company that offers its customers access to the Internet. The ISP connects to its customers using a data transmission technology appropriate for delivering Internet Protocol datagrams, such as dial-up, DSL, cable modem, wireless or dedicated high-speed interconnects." http://en.wikipedia.org/wiki/ISP

    I know it is illogical and counter-intuitive but actually they were right. ISP provides access to sites (they 'provide' you with sites ;) ) not the hosting (or rather not mainly).

  • IT Girl (unregistered) in reply to Gio
    Gio:
    I always thought tri-state meant NY, NJ and CT: http://en.wikipedia.org/wiki/Tri-State_Region

    Apparently you didn't read it very closely. "The Tri-State Region around New York City is where New York, New Jersey, and Connecticut nearly meet."

    Wikipedia distinguishes "tri-state region" from tri-state area", the latter apparently applying to any area where three states actually converge to a point.

    Not being American (but living very close to the border), I always thought tri-state area was as general as "greater metropolitan area" when used to generically reference, as opposed to something specific like GTA (Greater Toronto Area)

  • Zygo (unregistered) in reply to Bram
    Bram:
    Backup? On a $75/mo package? Mwahahahaha.... Sorry, Dogbert moment.

    The usual ISP deal at that price level is that they provide space for you to make your files available to the internet, you're responsible for having backups &c. At additional cost they may be able to offer you a backup solution as well. The fully hosted, supported, backed up applications providers tend to be "a bit" more expensive.

    I have a $70/mo package which includes (among many other things) 1GB of backups. For extra cost I can get more backup space, more throughput per day, etc.

    I, the customer, am responsible for configuring and using this service properly...which means it wouldn't have helped MegaPetCo.

  • (cs) in reply to someguy
    someguy:
    what morons, if you've got a customer who's shelling-out an extra $4,000/mo in fees despite cheaper alternatives which you have offered, cling to that customer for dear life! Upgrade his plan behind the scenes, back up his files regularly, hire someone to keep an eye on their site full-time and /still/ wind up $2000 better for it. If you couldn't keep their site up despite having $4000 in free money every month, you really have only yourselves to blame.

    Seriously, what an absolutely worthless company.

    I guess that's one way of looking at it... never really considered that. When someone insists on throwing boatloads of money at you, don't complain.

    Note: 4k+ per month is enough to get several full racks worth of space, power for all of them, and connectivity from multiple ISPs. It's really too bad they didn't realize that.

  • Zygo (unregistered) in reply to Americium
    Americium:
    Transaction management still has its limits. The database server has to track all the changes in some buffer. In Oracle, this is referred to as the ROLLBACK SEGMENT.

    DELETE FROM THE_TABLE; requires a ROLLBACK SEGMENT holding all the rows. A GB database is too large to handle, period.

    Only if you're running Oracle, whose implementation is replete with fixed size limits and uses in-memory data structures to implement transactions. Other database engines have no trouble executing this sort of operation (although in this specific case, TRUNCATE will be much faster).

    Well, OK, the delete will hold a lock on every individual row in the table, so you might get a deadlock if other clients are modifying rows in the same table...but if you're lucky, the deadlock will abort all the other transactions and not the DELETE FROM TABLE one. ;)

  • Dover (unregistered)

    Don't forget Florida, Alabama, Georgia. Oh, and South Dakota, Wyoming, Nebraska. And Oklahoma, Arkansas, Texas. And... heck, just look for the corners: http://maps.google.com/

  • Franz Kafka (unregistered) in reply to p2kmon
    p2kmon:
    I believe the story. I worked at a Microsoft for awhile and we had a VERY, VERY large telecom call up and they lost some major harddrives. They had Backups for 6 years.

    Problem is that NO one EVER tested the backups. So even if a backup is done and you don't test them, you could be in a whole lot of problems. Not a single backup worked.

    Um, if you don't do restores regularly, you don't have backups.

  • monkeyPushButton (unregistered) in reply to dkf
    dkf:
    Maybe they kept their backups in more columns of the database...
    Thanks. Eating split pea soup for lunch and nearly did a Linda Blair over my monitor.
  • BK (unregistered) in reply to d.k. Allen
    d.k. Allen:
    If I committed to backing up their data, and a restore failed for any reason, my customer is going to sue me to death. I'm for sure gonna charge extra for that risk. OTOH, All data would be RAID'd, certainly. Not a backup plan, but protection against failing drives.

    ditto

Leave a comment on “Death by Delete”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article