• (cs) in reply to tekiegreg
    tekiegreg:
    Duke of New York:
    Didn't WorldCom and Enron file only a few days after their scam was found out?

    Yeah but unlike these guys their financial systems were running to the end. With Enron, I'm fairly certain the real numbers were known by someone at the place. The inmates may have been running that asylum but there had to be one figure of sanity somewhere...

    Sanity had little to do with Enron. Yes, there was someone at Enron who knew exactly how much they didn't have - to the penny. Exceedingly meticulous. However, I can point to you many people at an OCD conference who are exceedingly meticulous, but not sane.

    If the financial type at Enron had both knowledge of how much money Enron did not have and sanity, he would not have been at Enron. A sane individual would have fled that company years earlier. (I seem to recall that there had been such a person, in fact.)

  • (cs) in reply to snoofle
    snoofle:
    I'm surprised they didn't follow the patterns of the one thing Wall Street actually got right! A few years back there was a massive power outage that took out most of Manhattan, and getting a cab to our backup site in NJ wasn't practical in the timeframe available. Buildings went dark, and a few of the main UPS units blew out so everything was off line. But the phones still worked. They just went back to doing everything with paper order slips and let an army of clerks sort it out after the fact. And these are exceedingly high volume businesses. I can't imagine a pet supply store had that much transaction volume...

    They probably did. However, when the cash registers no longer know any item prices, and the items have no price tags, just bar codes, everything is a price check. Customers won't put up with that for long.

  • Idler (unregistered) in reply to SR
    SR:
    Junkie:
    "The database was incredibly simple: a single table with hundreds of columns."

    Noone has a problem with this?

    Captcha: populus - man, I used to love that game!

    It's simple. Lunacy but simple.

    No, it's not lunacy. It's wtf ... errr ... concept known as unitabular database architecture. Running very strong with some managerial types floating on the fringes of IT. And, no, this isn't a joke :D

  • (cs)

    OK altogether now: "The REAL WTF is...."

    Seriously though, there's no way that one database crash alone caused bankruptcy...it musta been f***d up in the first place. Remeber pets.com? they went out with a whimper, not a bang....

  • Mike (unregistered) in reply to blindman
    blindman:
    10mb for a database? You must be joking. I wouldn't store my grocery list in a 10mb database.

    You must be one hungry bastard.... 10MB would hold a hell of grocery list.....

  • (cs) in reply to Anti-virus is a scam
    Anti-virus is a scam:
    No, this is why you NEVER EVER WRITE A DELETE QUERY.

    Seriously, if you have an application that uses a database, as opposed to "I store everything in the database and write everything, SELECT, INSERT, UPDATE, DELETE, manually. I really love how efficient the mysql command-line client is for running a business!", there is no reason you should ever be running a DELETE right in the database.

    Your application should have a delete_item() method which calls a delete_row() method which calls the actually DELETE, and all of that should generate in such a way that a WHERE-clause is always present.

    Special bonus: SQL has shitty syntax anyway, so you now avoid that.

    Great. Now as a programmer you come along and decide to change the database. Clean it up. Maybe delete a column that is no longer needed. So you have to write a program just to delete the column?

  • Duke of New York (unregistered) in reply to chrismcb
    chrismcb:
    Great. Now as a programmer you come along and decide to change the database. Clean it up. Maybe delete a column that is no longer needed. So you have to write a program just to delete the column?
    A programmer who just "comes along and decides" to "clean up" a database without clearing it first should be fired as quickly as possible.
  • Sylver (unregistered)

    This story is probably fabricated. Too many things don't make sense:

    First, the price of the hosting: $75 per month for 100 Mb of disk space???

    That could have made sense 10 years ago, but not now. I pay about $100 per year for a much better hosting, with 24/7 support, backups and whatsnot.

    Next, is the statement "For starters, its database ballooned to several gigabytes within the first month. On top of that, its Web traffic was far higher than the company's allotted bandwidth and other Web sites on the shared server were getting bogged down."

    That doesn't hold water. If the database was as described, it would NOT have ballooned to several Gbs within the first month. Either it would have been bigger from the getgo, or it would not have increased from less than 10 Mbs to several Gbs in a month.

    Next the idea that store employees putting store data in the database would bust the bandwidth ceiling isn't very realistic.

    Next, the database was loaded on the ISP's server, which means that another reasonably recent copy existed. Did it magically disappear?

    Those and other clues suggest the story is a fake.

  • Kyle (unregistered) in reply to dpm
    dpm:
    {many pointless guesses}

    Good heavens, am I the only one here who appreciates the clever show on the Disney Channel? As Doctor Heinz Doofenshmirz explains in almost every episode, the "Tri-State Area" is where Phineas & Ferb live.

    Seriously, it's a great show: the animation, the voices, but especially the writing. I heartily recommend it.

    Quoted for agreement!

  • Roger Garrett (unregistered) in reply to josephdietrich
    josephdietrich:
    Holy moly. That's not death by delete. That's death by stupidity, with an assist by delete.

    Deletes don't kill databases, people kill databases.

  • Jimmy (unregistered)

    Had similar experience once with my friend who accidentally deleted production data of customer. Thankfully the backups saved the disgrace. The Account Manager was furious on him, but mistakes do happen. The important part is to minimize the impact of mistakes !! Companies who are lazy / ignorant about backups are bound to a calamity some day, only the day has to come.

  • Jimmy (unregistered) in reply to Jimmy

    While deleting data on prod server, it is better to use a transaction context, so a rollback can be done if needed.

    Any delete queries should be run on stage first to make sure it is just deleting what it is supposed to delete.

    A simple where clause miss is going to wipe out the table !! So I would do 10 checks before doing anything with such a serious effect.

  • Jimmy (unregistered)

    The backups can only solve the problem to certain extent.

    As in clustered - distributed DB scenario, the backup is not that real-time. Depending on how frequent auto backup are taken, there is a likelihood of some delta getting missed post recovery.

    Also the time spent in finding the backups & recovery can be so huge depending on DB size, it might be worth investing time in doing peer code review before doing a DELETE !!

    Having said that, I would really hate to say that modern DB must be smart enough to cover the stupid developer ass! I usually love to hate the machine than the human.

  • DavidM (unregistered) in reply to synp
    Only in well regulated industries [...] financial industry.

    Are we talking about the same people here ? Oh, that was irony ? ;)

  • acid (unregistered) in reply to blindman
    blindman:
    10mb for a database? You must be joking. I wouldn't store my grocery list in a 10mb database.

    1. I suspect you need to go on a diet. 10Mb = ~20 novels in plain text. Even counting DB overhead and poor table structure, let's say you only have 10% of that space used for actual data, that's still one hell of a shopping list.

    2. Think Gloves. A RDBMS is a scoche of overkill for a shopping list. I use a pen and paper. This has the added advantage of my boys seeing me write things down, deomonstrates that writing is useful in a world where that's not as obvious anymore.

    3. Your point is still valid though. For what they were going to use it for, the DB was VERY poorly specced. One wonders whether or not the original advice came from our Delete Monkey?

  • acid (unregistered) in reply to ounos
    ounos:
    The code monkey survived that? They didn't hang him upside down from the ceiling?

    It was a pet store. They have inspectors. You can't just hang monkeys upside down from the ceiling, no matter how stupid they are.

    Personally I've been on the receiving end of more than one stupid delete statement/command initiated by people who should know better. The person who should've been sued in this case was the monkey. We're asked to carry professional indemnity for a reason folks.

    CAPTCHA - Nulla - final size of MegaPetCo's database?

  • acid (unregistered) in reply to Junkie
    Junkie:
    Captcha: populus - man, I used to love that game!

    I still do. dosbox.sourceforge.net

  • (cs) in reply to Sylver
    Sylver:
    This story is probably fabricated. Too many things don't make sense:

    First, the price of the hosting: $75 per month for 100 Mb of disk space???

    These things rarely say exactly when they take place. I just assumed that it was long enough ago for the price to be... high but within the normal range for low-end.

    Sylver:
    Next, is the statement "For starters, its database ballooned to several gigabytes within the first month. On top of that, its Web traffic was far higher than the company's allotted bandwidth and other Web sites on the shared server were getting bogged down."

    That doesn't hold water. If the database was as described, it would NOT have ballooned to several Gbs within the first month. Either it would have been bigger from the getgo, or it would not have increased from less than 10 Mbs to several Gbs in a month.

    I'd not be surprised if it started out being just the database for the website, but things got moved over to it. "Why are we paying for this database when we have a perfectly good database at the web hosting company. We need that one, because they can't use our local SQLite db to run it. We don't need the local one."

    Sylver:
    Next the idea that store employees putting store data in the database would bust the bandwidth ceiling isn't very realistic.

    Think camels and straws. Think poorly designed protocols. Think 'select * from db;'

    Sylver:
    Next, the database was loaded on the ISP's server, which means that another reasonably recent copy existed. Did it magically disappear?

    The only copy was the ISP's server. When that died, it all died.

    Sylver:
    Those and other clues suggest the story is a fake.

    Most of the clues people use to determine stories here are fake fall into two categories:

    1. Stupid things people do that other people think are too stupid for people to actually do. But we know people do these things.

    2. Details which are probably anonymisation artifacts.

    I've seen people do most of the stupid shit in this story - just not all on the same project.

  • European (unregistered) in reply to aaawww
    aaawww:
    benelux?

    But rhat would be tri-country, not tri-state.

  • manwhore (unregistered)

    After reading that, I skipped right past saying both 'WHAT', and 'THE' .. and headed straight for the 'F" word.

  • jay (unregistered) in reply to Zemm
    Zemm:
    jmzrbnsn:
    New South Wales, Victoria, South Australia, or New South Wales, Queensland, South Australia?? The closest Tri-State Area round these parts....

    Should be "NSW/QLD/SA" since if we are supposed to decipher "NY/NJ/CT" they can decipher our state codes too! Though in a country with only six states and requiring 3-letter state codes it a minor WTF...

    Hey, good point! Couldn't Australia get along with one-letter state abbreviations?

    Reminds me of a panel discussion I attended once comparing American to Australian culture. One of the Americans commented that she thought it was interesting that Australia has about the same land area as the United States, but the United States is divided into 50 states while Australia is divided into only 6. To which an Australian replied, "Yes, but of course you don't have 50 separate state governments, do you?" Well, uh, yes, actually we do ...

  • RoverDaddy (unregistered) in reply to jay

    I ran into this when I was in college in upstate N.Y. (near Albany). We ran a bus service for Thanksgiving and spring break, and people would come by looking for tickets to 'the city' (Huh?) I needed somebody else to explain to me that they meant NYC. But what can you expect from people who think Westchester County is 'upstate' when even from Albany, there are almost 180 miles MORE upstate to go 'til you reach the Canadian border.

    captcha: opto - reminds me of opto-isolators used in factory control circuits.

  • (cs) in reply to Ren
    Ren:
    Tri-State has to be TRUE, FALSE, FILENOTFOUND. That's the only thing that makes sense.

    The ISP lost a huge cash cow here. The managers of PetCo were (obviously) technically incompetent, but that sort of people tend to appreciate bullshit when ladled up.

    Here's what should have happened when the slowness reason was discovered:

    1. Backup everything. Use your personal space if need be.
    2. Offer to PetCo analysis of their database. If they refuse, go to 8
    3. After two weeks (and, say $3,000) you report to them that their system is outdated and offer to them a shiny bidoof! Er, new system that works twice as fast!
    4. Split the WEBSITE data to its own (properly indexed) table. After two weeks (and, say, $8,000) you present the new system. The website is fast, people are happy.
    5. Offer to do the same for <part X> of their database.
    6. Split <part X> to its own table (properly indexed) and after two weeks (and, say, $10,000) present the new system.
    7. Repeat 5-6 until you run out of parts
    8. When their database inevitably collapses, offer your services (at, say, $2,000). Also offer analysis at a discount (say, $1,000). Restore backup, system is running. Go to 2.

    Say that again. So you have a customer with a moronic database, and because of that setup they are paying you $4000 extra a month. So you are going to offer to fix the system for them? eh?

  • sarails (unregistered)

    Like others have said, a back up is not a backup until it's been tested; I once worked for a small company which had the SME market as it's primary focus. This one time I was able to gain a contract for a boutique wine maker, of international reputation, to upgrade some PC's and the storage on their server. Thinking that we had a great start, as they had been running Tape backups of their data "for ever" and RAID 1 on the drives in the server running Windows NT.

    We tried to test the backups on another machine - FAILURE nothing on the tapes.

    Still thinking that all was OK as there was the RAID 1 partition, my boss, went to upgrade the operating system in the machine (going to Small Business Server), and being the type of person that he was, FDISK is the only way to start. Whilst he was getting ready to kill the partition, I demanded that he check the RAID backup "for safety". With much cursing and grumbling he asceeded to my demands to check the alternate partition. good thing we did as FAILURE dud drive on the alternate partition, it had never been setup correctly by the previous IT company.

    I never even got a thanks for saving the business from making a major fubar to the customers files.

    All being said I left there not too long afterwards, and the IT compay closed it's doors a while later.

    The boutique winery, did end up getting a Server that did create tape backups, and a RAID system that worked too.

  • Anon (unregistered) in reply to Lucus

    The government should act in everybody's interest. Here we see hundreds of jobs lost to stupidity, not to mention an unknown amount of tax dollars.

    So a regulation that requires backups only propagates knowledge gained from experience: Avoid job losses by having good backups. More jobs and lower taxes mean more happy voters, too.

    And that regulation could also take some wind (cash) out of the managers' golden parachutes by imposing stiff penalties. At that point the difference between $75 and $4000 suddenly stops looking so enormous.

  • acid (unregistered) in reply to Anon

    Ummmmm....

    Hang on a bit.

    Anon:
    The government should act in everybody's interest. Here we see hundreds of jobs lost to stupidity, not to mention an unknown amount of tax dollars.

    The Govt SHOULD act in the people's interest, but then so should the people. I don't know about you but I don't want to live in a Nanny state. I don't want a Govt inspector in my toilet stall at work watching me wipe my lily white muscular arse in case I miss a spot. I should do a good job, for the sake of my health and for the olfactory sake of my work colleagues. That's enlightened self interest.

    Anon:
    So a regulation that requires backups only propagates knowledge gained from experience: Avoid job losses by having good backups. More jobs and lower taxes mean more happy voters, too.

    The trouble with regulations is that they need to be enforced. That actually INCREASES taxes, it doesn't reduce them. Good management and good business sense (in fact good COMMON sense) tells me that MegaPetCo should have been keeping backups. They weren't. This is not the Govt's fault, it's the fault of the executive of MegaPetCo for not ensuring their business was safe from 'disaster'. Business Continuity is a part of any business plan these days and it has to be - it should be. Part of any BC plan is working and tested backups.

    Getting the Govt to intrude on something that 99.9% of the business community either is or thinks they're doing seems like another itrusive waste of taxpayer's money.

    Anon:
    And that regulation could also take some wind (cash) out of the managers' golden parachutes by imposing stiff penalties. At that point the difference between $75 and $4000 suddenly stops looking so enormous.

    Well this is the real problem that you're trying to solve here. The managers shouldn't come out of this scott free after their fundamental mistake has destroyed their business. Your way of doing it however simply doesn't hold water.

    Here in Australia we're working on ways to stop executives from bailing out with massive payouts as companies go bankrupt. We've had the odd spectacular failure (especially in the insurance sector) over the last decade, and executives are being held more accountable for the way they conduct their businesses.

    BUT, how far do you go? Do you have Govt auditors with caseloads, every business larger than X or every director with a salary larger than Y has to be periodically audited for how they run their business dealings? I mean, why stop at backups? Why even START there?

    Businesses fare best when the intervention from Govt is less, not more. Occasionally someone gets it wrong and there are massive consequences because businesses are so large. That's not malicious, it's business. Some people make profit, some people make losses. It's sad, but some people lose their jobs.

    My point is this - businesses are NOT social welfare programs. They're there to earn a profit. You work for one to help it earn a profit. Generally they'll keep you employed for as long as what you do is more valuable to them than the cost of retaining your services. When a business fails, the people own the business are losers too. Your argument above seems to imply that the only people who lose from a failed business are the employees and that simply isn't true.

    So, should the Govt FORCE business to do something that almost everyone does simply because it's good business sense and stops things like MegaPetCo debacles? Well, no. They shouldn't. I don't want my tax dollars increased to fund the Inspectors for Doing the Obvious (IDO) Dept.

    What SHOULD the Govt do in cases like this? Put more funding into education. Teach people to not only to learn their curriculum, but teach them to think at the same time. Take the lesson from MegaPetCo, and feed it into the MBA course for every university in the land.

    Yes, MBA, not computer science. It's often been said that you don't need to be technical to manage technicians. We all know this to be wrong, but what we've lost sight of is that the reverse is now true - you have to have a measure of technical knowledge to be a mainstream manager of note in any business.

    Anon, I see and share your pain, but we need the Govt to fix the problem at the source - education, not regulation, is more likely to stop MegaPetCo from happening again.

  • linuxpaul (unregistered) in reply to acid
    acid:
    Anon:
    The government should act in everybody's interest. Here we see hundreds of jobs lost to stupidity, not to mention an unknown amount of tax dollars.
    Anon:
    So a regulation that requires backups only propagates knowledge gained from experience: Avoid job losses by having good backups. More jobs and lower taxes mean more happy voters, too.

    Here in Australia we're working..

    Anon, I see and share your pain, but we need the Govt to fix the problem at the source - education, not regulation, is more likely to stop MegaPetCo from happening again.

    First of all, its "I feel your pain"(tm).

    Second, the US is slowly drifting to a consensus consistent with Anon's POV (see: slashdot). If I were you, acid, I would prep myself for a sudden influx of the productive US citizens as immigrants. Some smart tax policy, and Australia is poised to become the next economic super-power. We'll leave the US to the the Anons of the world to figure out how to pay off $10T in debt by complying with their federally-mandated backup policies.

  • (cs) in reply to Tweet
    Tweet:
    The only downside is the innocent employees.
    ...and the "stock".

    And the local chapter of the Humane Society and all the no-kill shelters that have to pick up the pieces when a bunch of breeders suddenly have nobody to sell to.

  • Nimrand (unregistered) in reply to jay

    Yes.

    And I also need a law that tells me not to break the law so that I wont go to jail for stupidly ignoring all these laws.

  • (cs)

    I have to say they deserved to go out of business. Anybody who takes IT operations and development lightly in this day and age should not be managing a company that relies on technology for its survival.

  • Ima Techie (unregistered) in reply to wbo

    Having been admin at 3 different ISPs, we've ALWAYS had regular backups, at least weekly. So there are two WTFs:

    1. Company has a "ninja" developer.

    2. ISP has horrid service.

  • Unca Alby (unregistered)

    HEY BOSS! I GOT GOOD NEWS AND BAD NEWS!

    The good news is, I solved the space problem on the disk ...

  • mambo (unregistered) in reply to cfreak

    "TRWTF is that the ISP had no backups. They deserved to be sued. It isn't their place to determine if a customer's database is crappy. Even if the $75 account meant no backups were made, the developer should have made backups before any work was done!"

    The real WTF is playing with a production system without first trying out on a mirror... That being said: What in heaven makes you believe a webspace provider should make any backups? I'd sue anyone making backups of my data without my permission because it's potentially deadly stuff we talk about: wages, debts etc. Let some of those things leak to the public and it could have a devastating effect on any business sending their stocks reight through the bottom -.-

    On the other hand I can tell you exactly how that database came to be: a fast expansion of the business and no own IT department (if they had one it was propably "Larry in the basement" who was responsible for kicking the desktops when windows wouldn't boot). After some time they need some new feature for accounts so they hire the cheapest guy they can get. That guy is either an idiot or well underpaid and lazy and just adds the stuff to the existing table. Four or five of those guys later you need a fucking genius to identify which column belongs to which value. Plus you have the php code of some of the pages somewhere hidden in the table. Nobody really wants to mess with this (at least not at the price they are willing to pay) and they don't have the knowledge to see why this is bad ("It works."). In the end nobody is responsible: the company didn't have a clue about backups, the developers didn't get paid well enough to give a shit and the ISP didn't get paid to make backups.

  • drobnox (unregistered) in reply to bosox1256

    I grew up in range of both New York and Philadelphia TV stations, and was included as part of two tri-state areas: NT/CT/NJ and PA/DE/NJ

Leave a comment on “Death by Delete”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article