• jhkjh (unregistered)

    I don't get the people who say $75 a month shouldn't include backups. I pay $4/month get 100GB/month in bandwidth, 3GB of space, as well as backups, although if you need to access the backups, you have to pay a service charge unless it is their fault (i.e server hdd dies)

    Unless this article was from 10 years ago, then I understand.

  • Zeal (unregistered) in reply to merpius
    merpius:
    Just for the record, Oregon, Washington and Idaho is NOT called the tri-state area..... you know, just in case you wanted to know another useless piece of trivia.

    You apparently haven't been to the Idaho panhandle. There is a sports store there called "Tri-State" which I believe refers to these 3 states.

    In fact, pretty much any 3 bordering states could be called this, which is every state except for around the borders, and the "Four Corners" part of Colorado, Utah, New Mexico, and Arizona.

  • Craig Lewis (unregistered) in reply to Yeah
    This is why you "start transaction" before deleting 10 million rows.

    This is probably before MySQL supported transactions. Because, you know, a fast delete is better.

  • Jeff (unregistered) in reply to Smash King
    Smash King:
    St Mary's Hospital for the Non-Euclidean Geography:
    Hmmm, where is the Tri-State Area?

    I thought of Hawaii. But swimming in any direction, you run ashore in either California, Washington and Oregon. And Alaska.

    Does anybody have a clue?

    Solid/Liquid/Gas

    clap BRILLIANT Smash King.

  • Big G (unregistered) in reply to kastein
    kastein:
    someguy:
    what morons, if you've got a customer who's shelling-out an extra $4,000/mo in fees despite cheaper alternatives which you have offered, cling to that customer for dear life! Upgrade his plan behind the scenes, back up his files regularly, hire someone to keep an eye on their site full-time and /still/ wind up $2000 better for it. If you couldn't keep their site up despite having $4000 in free money every month, you really have only yourselves to blame.

    Seriously, what an absolutely worthless company.

    I guess that's one way of looking at it... never really considered that. When someone insists on throwing boatloads of money at you, don't complain.

    Note: 4k+ per month is enough to get several full racks worth of space, power for all of them, and connectivity from multiple ISPs. It's really too bad they didn't realize that.

    I wonder if they didn't realize they were paying $4k/month? I could see the $75/month in one person's budget and the $4k+ from some other account. Therefore, the decision-maker thought he was only spending $75/month!

  • (cs) in reply to hikari
    hikari:
    dpm:
    I've never run an ISP, but I do host domains, and I certainly make backups of all the files. I would expect a real ISP to do the same, but apparently that is not the case here.

    I would expect a real business to backup its own data, apparently that wasn't the case here either. Along with a few other cases. Like not hosting your company's entire IT infrastructure on a remote server.

    I'm thinking of what happens if I screw up (hard drive crashes, I mistakenly wipe out a filesystem, or even just one file gets deleted by accident). If I cause the problem, I would think that I would be liable to some degree; therefore, I make backups of everything for my sake.

  • Shinobu (unregistered)

    And people don't believe you if you say there companies' data is its most important asset. Wow. Also, overcharges are what they are to finance emergency service increase. Don't just treat them as fines. Essentially, both companies lost out big time due to detrimental savings (getting money by forgetting to invest in a long-term necessity), one by opting to stop properly investing in its IT infrastructure and the other by pocketing the overcharges instead of investing them. The effects for the client company were perhaps more dramatic than for the hosting company, but it also missed out on a lot of money.

  • n (unregistered) in reply to Americium
    Americium:

    Transaction management still has its limits. The database server has to track all the changes in some buffer. In Oracle, this is referred to as the ROLLBACK SEGMENT.

    DELETE FROM THE_TABLE; requires a ROLLBACK SEGMENT holding all the rows. A GB database is too large to handle, period. The transaction fails due to memory or disk space limits.

    The poor developer couldn't rollback a transaction!

    Couldn't roll back the transaction? I thought the point of transactions (and ACID compliance in general) was if everything that was requested to be done was not able to be done (out of memory, diskspace, etc), the transaction would abort automatically leaving the database in a consistent state.

    So in your case would the DBMS would start deleting the data then when it runs out of memory for it to continue recording the deletes it would just automatically abort without the user needing to do anything.

    That being said I have never used Oracle, but I am pretty sure it is ACID compliant.

  • Coward (unregistered) in reply to blah

    Were talking front page material here folks:

    blah:
    You idiots! The tri-state area is obviously enum { True, False, FileNotFound }!
  • (cs)

    While it's sad that people got laid off, at least that's one less idiot in business. Clueless cheapskates deserve to be run out of town until they learn that you need to spend money if you run a business - trying to coast along spending as little as possible is not the way real businesspeople work.

  • (cs) in reply to cfreak
    cfreak:
    TRWTF is that the ISP had no backups. They deserved to be sued. It isn't their place to determine if a customer's database is crappy.
    If it's not written in the contract, they can't be sued over it. Think: who actually deleted the data? An authorized employee of the store, NOT anyone at the ISP.
    cfreak:
    Even if the $75 account meant no backups were made, the developer should have made backups before any work was done!
    {golf clap}

    Truly, your superhuman insight has penetrated to the core of the troubles: the owner of the data is responsible for ensuring that there is a copy safely stored. Good thing we had you here to point that out to us!

  • (cs)

    {many pointless guesses}

    Good heavens, am I the only one here who appreciates the clever show on the Disney Channel? As Doctor Heinz Doofenshmirz explains in almost every episode, the "Tri-State Area" is where Phineas & Ferb live.

    Seriously, it's a great show: the animation, the voices, but especially the writing. I heartily recommend it.

  • My Name (unregistered)

    This is probably the only time I have read an article that deserves this, so I'll write it now:

    E P I C F A I L!

    There, I've finally found a reason to write it. The internet has won.

  • Bluesman (unregistered) in reply to SlyEcho
    SlyEcho:
    Yeah, but what happened to all the animals?

    Alas, they too were stored in the database...

  • JohnB (unregistered) in reply to Dennis
    Dennis:
    Don't forget ArkLaTex!
    OK LA ho MA.
  • (cs) in reply to Coward
    Coward:
    Were talking front page material here folks:
    blah:
    You idiots! The tri-state area is obviously enum { True, False, FileNotFound }!
    To REAL engineers, tri-state is Low, High, Hi-Z. And VHDL/Verilog go even further with their bit value enums :)
  • guy334 (unregistered)

    The NY Tri-state area is: Long Island/NYC/Hudson Valley; Northern NJ;Western CT

  • Zygo (unregistered) in reply to Shinobu
    Shinobu:
    Also, overcharges are what they are to finance emergency service increase. Don't just treat them as fines.

    The original article did mention that the "shared" hosting server was actually dedicated to the one customer, so the charges were being applied to increase service. The problem in this case is that no amount of increasing service capacity would have changed the fact that no backup service was being provided at all.

    Also from the article, the sales people were trying to contact the customer and get them onto a more appropriate service package, but the customer a) didn't want to discuss the option, and b) was willing to pay the overcharges month after month. Maybe if I were in that situation I'd ask the customer to sign a waiver that states that they had been offered a better deal and declined it, in case the customer comes to his senses some day and sends their lawyer after me.

    The ISP might have upgraded the server, or started making backups of the entire server (a few of those monthly payments would have bought a dedicated tape changer, a few more would have bought a SAN box with snapshot capability), but given the site's multi-second page load times, adding backups to that load might not have won any favor from the customer. How do you sell backups to a customer when you can't sell an order-of-magnitude reduction in their monthly payment combined with improved service? Also, how do you deploy this custom solution for only one customer without creating a developmestruction environment in your own shop?

    After the ISP's sales guy offers to waive the overcharges if the customer changed to a more appropriate--and cheaper--service package, and after the hardware is purchased to dedicate a "shared" server to this single customer, any further charges are fines. No one is meant to pay overcharges over the long-term--that's why they're priced so far above not just reasonable market rates, but the rates of equivalent alternative services from the same provider.

  • (cs) in reply to JdFalcon04
    JdFalcon04:
    Ickithus:
    Ok, I'll add one too, Evansville Indiana. Indiana, Kentucky, Illinois. Seriously, there are a lot more tri-state areas than you'd think.

    I'll throw one in as well... New Hampshire, Mass, Vermont. They have (had? it's been a while) a "tri-state lottery."

    Tri-State Lottery is Maine, New Hampshire, and Vermont, but other than that, I've never heard them referred to as "tri-state". They're normally called Northern New England, or sometimes "those states up north with a lot of trees and more moose than people" by people from Southern New England.

  • Procedural (unregistered) in reply to cfreak
    cfreak:
    Wow ... I'm going to have to call BS on this story.

    First of all ... not knowing how much money that had? WTF? One would assume they have bank-accounts. That typically gives one a good idea. Its not like their bank was storing records there. That could have given them plenty of information on the amount of money likely lost and a basis for a lawsuit.

    Their vendors probably would have been happy to reprint invoices if some were unpaid and missing.

    The biggest problem would be is if they had unpaid invoices ... but if they were a retail chain its not all that likely they would have had many.

    TRWTF is that the ISP had no backups. They deserved to be sued. It isn't their place to determine if a customer's database is crappy. Even if the $75 account meant no backups were made, the developer should have made backups before any work was done!

    I suspect that if this story is true at all there's a lot more to it than what is here

    Yeah, no. That works if you sell lemonade, but to assume that what's in the account represents what you can spend is actually worse business sense than having no backups at all and setting your server over the thermo-pump and under a leaking roof. It's called accrual accounting because biz types needed to give it a nice name, but otherwise it's called shoebox accounting.

    Some money is in your acocunt in transit; some is owed to suppliers, employees, pension funds, etc. Cash on hand != Spendable cash.

  • Bob (unregistered) in reply to Leo
    Leo:
    GT:
    Thank you. There is a cute little "tri-state gas station" in Port Jervis where NY, PA, and NJ meet, but come on people, I can't believe anyone would think of that.

    Unless you are talking about NY/NJ/CT, you have to qualify your "tri-state area".

    Why would you not have to qualify it then? Oh, right, because NYCers think they're the center of the universe. Sorry, but most people think it's a hellhole.

    And you always have to qualify your tri-state area unless you're using it in an obviously local context to other local people.

    Um, no, as I learned when I moved, you must ALWAYS qualify to outsiders. NY/NJ/CT does not come to mind first for me. When I am in Eastern PA, tri-state is eastern PA/NJ/NY. When I am in Pittsburgh, it is Ohio/WV/Western PA. Get out of your little cultural box. The country doesn't revolve around NYC.

  • Dwayne (unregistered) in reply to Tweet
    Tweet:
    In short, this company deserved to fail -- no, needed to fail. And the sooner the better. With no bailout option. The only downside is the innocent employees.
    Think of it as evolution in action.
  • movzx (unregistered) in reply to cfreak
    cfreak:
    TRWTF is that the ISP had no backups. They deserved to be sued. It isn't their place to determine if a customer's database is crappy. Even if the $75 account meant no backups were made, the developer should have made backups before any work was done!

    The developer worked for MegaPetCo, not the ISP.

    If you do not pay for a backup service do not expect backups.

  • ForcedSterilizationsForAll (unregistered) in reply to Americium
    Americium:
    Satanicpuppy:
    It's a php website. So we're talking MySQL databases. The whole database is a single table, and MySQL still defaults to MyISAM instead of INNODB, so you probably need the sort of skill that you'd normally use to normalize a database to even know about transactions and that not all mysql tables are transaction capable by default.

    So, in conclusion, what transaction safe database engine do you think they're using?

    The fall of the company had nothing to do with its database, and everything to do with its crappy management.

    Transaction management still has its limits. The database server has to track all the changes in some buffer. In Oracle, this is referred to as the ROLLBACK SEGMENT.

    DELETE FROM THE_TABLE; requires a ROLLBACK SEGMENT holding all the rows. A GB database is too large to handle, period. The transaction fails due to memory or disk space limits.

    The poor developer couldn't rollback a transaction!

    This is why you write your WHERE statements before your FROM in deletes and your SET statements in your UPDATES. I've had my own share of OMGIREALLYSCREWEDUPONTHISANDAMPROBABLYGOINGTOGETFIRED hyperventilating moments. Part of it was because I wasn't aware of the begin/commit transactions at the time since I was a wet behind the ears developer at the time.

  • ForcedSterilizationsForAll (unregistered) in reply to jhkjh
    jhkjh:
    I don't get the people who say $75 a month shouldn't include backups. I pay $4/month get 100GB/month in bandwidth, 3GB of space, as well as backups, although if you need to access the backups, you have to pay a service charge unless it is their fault (i.e server hdd dies)

    Unless this article was from 10 years ago, then I understand.

    You think 10 megs for a database is enough by today's standards? Those numbers alone told me this was probably from 10 years ago.

  • Peter (unregistered) in reply to Dover
    Dover:
    Don't forget Florida, Alabama, Georgia. Oh, and South Dakota, Wyoming, Nebraska. And Oklahoma, Arkansas, Texas. And... heck, just look for the corners: http://maps.google.com/
    I have great difficulty in understanding why so many people seem to be interested in the multiplicity of definitions of "tri-state area". Before encountering all these postings, I neither knew nor cared what the phrase meant. I still don't care.
  • (cs) in reply to dpm
    dpm:
    I've never run an ISP, but I do host domains, and I certainly make backups of all the files. I would expect a real ISP to do the same, but apparently that is not the case here.

    You're siding with the PetCo? Dude, WTF? They just hosted stuff. The pet company didn't want to move to a dedicated server, and they paid for services a la carte. If they didn't arrange to pay for backups to begin with, why would you think the ISP would just suddenly start making backups of PetCo's data? Would they just have done so out of the goodness of their heart?

  • nzt (unregistered) in reply to My Name
    My Name:
    This is probably the only time I have read an article that deserves this, so I'll write it now:

    E P I C F A I L!

    There, I've finally found a reason to write it. The internet has won.

    Did you by any chance meant to write "Epic Failure"?

    Can't really qualify this is epic though. I'd say the maiden voyage of the Titanic and the Tacoma Narrow Bridge are examples of failures of epic proportions.

  • Franz Kafka (unregistered) in reply to Zygo
    Zygo:
    After the ISP's sales guy offers to waive the overcharges if the customer changed to a more appropriate--and cheaper--service package, and after the hardware is purchased to dedicate a "shared" server to this single customer, any further charges are fines. No one is meant to pay overcharges over the long-term--that's why they're priced so far above not just reasonable market rates, but the rates of equivalent alternative services from the same provider.

    So what if they are punitive. It's not the hosting provider's fault that their customer is an idiot.

  • Tristater (unregistered) in reply to Bob
    Bob:
    The tri-state area could be a couple of places. Once possibility is NY/NJ/Eastern PA. Another is Ohio/WV/Western PA. I've even heard PA/DE/MD referred to as the tri-state area.

    KY/OH/IN is also a tri-state area.

  • Shea (unregistered) in reply to Ouch!

    No there is not a law to have backups - nor should there be. We also shouldn't need a law that tells people to not do incredibly stupid things, but freedom includes the right to do things... well... incredibly stupid; as long as it does not impinge on others rights or endanger others.

    Point is - we sure don't need laws telling us how to design proper technology. Either people do it competently, or they're out of business on a business ending note such as this.

    The real crime here is anyone thinking "IT infrastrucutre" consists of a $75 a month contract. They paid for their first small server farm the first month. After six months at $4k/mo, they should have had a complete tape jukebox, automatic failover and almost redundant everything (albiet at a cheap level).

    Corporate deserved the results - the regular rank and file did not :-/

  • jay (unregistered) in reply to Ouch!
    Ouch!:
    Isn't there a law requiring a company to have backups?

    Do you really need a law to prevent you from doing stupid things to hurt yourself? Do you need a law to tell you to change the oil in your car now and then, to keep the batteries in your smoke alarm charged, and not to gargle with drain cleaner? How would we hire all the policemen to enforce such laws? Would they visit your home or business regularly to check on you?

    The scary part is, we actually have many such laws.

  • lol (unregistered)

    Pro tip: the 48 contiguous states have an abundance of tri-state areas.

  • jay (unregistered)

    "Tri-state" refers to the logic in SQL: A boolean variable can be true, false, or null. All the logical operators use tri-state logic, e.g. T && T = T, T && F = F, T && null = null.

  • (cs) in reply to wee
    wee:
    dpm:
    I've never run an ISP, but I do host domains, and I certainly make backups of all the files. I would expect a real ISP to do the same, but apparently that is not the case here.
    You're siding with the PetCo? Dude, WTF?
    Feel free to read my other posts.
  • jay (unregistered) in reply to GT
    GT:
    Thank you. There is a cute little "tri-state gas station" in Port Jervis where NY, PA, and NJ meet, but come on people, I can't believe anyone would think of that.

    Unless you are talking about NY/NJ/CT, you have to qualify your "tri-state area".

    I hope you're trolling, but I wouldn't be surprised if you're not. When I lived in New York, it was common to refer to New York City as "the City", as if it was the only city in the world. I accepted this as normal until I moved away and discovered that the rest of the world does not divide the universe into "New York / New Jersey / Connecticut tri-state area" and "other places rumored to exist".

  • Zygo (unregistered) in reply to Bob
    Bob:
    The country doesn't revolve around NYC.

    Quite correct. The country, in fact, revolves around the core of the planet Earth; however, the core of the planet Earth just happens to be located under NYC.

  • jay (unregistered)

    Usuaul disclaimer of "I am not a lawyer", but if the ISP never claimed that they did backups, if it was not in the contract or the advertising, I don't see how they could be sued for failing to do backups. Like, gas stations used to have someone check your oil every time you came in for a fill-up. Now they don't. If my car runs out of oil and the engine blows up, can I sue the gas station for not checking the oil?

    Maybe the pet store can sue their gas station for not backing up their data for them.

  • Lucus (unregistered) in reply to Ouch!

    Why the hell would you think it's the government's responsibility to see to it that companies have backups???

  • (cs) in reply to Kazan
    Kazan:
    Ouch!:
    Isn't there a law requiring a company to have backups? If that story hasn't been exaggerated out of proportion, really, it's a miracle they lasted more than three months.

    and you know how many companies ignore regulations?

    Moreover, the plausible threat of having your entire empire come to an grinding, irrevocable halt is ample motivation for the vast majority of organizations. Those who fail to heed suffer the ultimate consequence and serve as an inherent warning. Why do we need the government to pile on and waste the time of those who are already acting responsibly?

    What makes you think that the hundreds of government employees who would do the regulating are any smarter than those they would regulate? MegaPetCo, as evidenced by their initial success, at least has the advantage of being helmed by those who's qualifications supersede 'once won a popularity contest'.

  • Duke of New York (unregistered)

    They waited a few months to file bankruptcy?

  • (cs) in reply to Peter
    Peter:
    I have great difficulty in understanding why so many people seem to be interested in the multiplicity of definitions of "tri-state area". Before encountering all these postings, I neither knew nor cared what the phrase meant. I still don't care.
    Peter:
    Why are people on the internet so damn pedantic and annoying?

    FTFY

    Zygo:
    Bob:
    The country doesn't revolve around NYC.

    Quite correct. The country, in fact, revolves around the core of the planet Earth; however, the core of the planet Earth just happens to be located under NYC.

    Thanks, I'm keeping that one...

  • (cs)

    Actually, the tri-state area is east Texas, west Texas and the panhandle.

  • random.next (unregistered) in reply to Code Dependent

    I always thought it was Zion, the land of Oz and Mordor... You learn something every day.

  • Duke of New York (unregistered) in reply to My Name
    My Name:
    This is probably the only time I have read an article that deserves this, so I'll write it now:

    E P I C F A I L!

    There, I've finally found a reason to write it. The internet has won.

    go back to 4chan

  • Just Some Guy (unregistered) in reply to JoPoser
    JoPoser:
    Nitpicking....

    They were an IAP that also hosted. an ISP does host by default.

    What's an IAP? AO-effin-L. Internet Access Provider; I'm an ISP but not an IAP. Can't tell you how many people think that ISP means getting online and are upset when they talk to me on the phone.

    I've been either actively or indirectly involved in the ISP business for about 12 years and this is the first time I've heard someone make that distinction. Google for "internet service" some time and see how many companies are willing to provide it to end users.

    Basically, you're inventing a definition and then pissing off potential customers for not knowing of your kookery in advance.

  • (cs) in reply to sep332
    sep332:
    Neil:
    Smash King:
    St Mary's Hospital for the Non-Euclidean Geography:
    Hmmm, where is the Tri-State Area?

    I thought of Hawaii. But swimming in any direction, you run ashore in either California, Washington and Oregon. And Alaska.

    Does anybody have a clue?

    Solid/Liquid/Gas
    Plasma?
    Bose-Einstein condensate?

    You guys are headed to the lab for a definition, you need to be headed to the marketing department.

    I worked for a small company that had to be licensed in three states because they had outlets in the only three metropolitan areas in the region - each of which was in a different U.S. State.

    The biggest problem with conducting business in that region was that customers routinely ended up in an outlet that was not in their home state, and would be unaware that they could return any purchased merchandise or get warranties serviced closer to home.

    If the company had been selling consumables it would not have been an issue, but trying to sell depreciable items to 'out-of-staters' is damn near impossible unless there is a sales tax benefit involved somehow.

    'Tri-state' is a consumer confidence thing, commonly used in America wherever three states meet.

  • Rich (unregistered) in reply to Yeah
    This is why you "start transaction" before deleting 10 million rows.

    any firm that has 10 million rows in one table, with different needs like accounting, billing, web pages all in one table, probably doesn't do much best practices.

  • Mu (unregistered) in reply to Duke of New York
    Duke of New York:
    They waited a few months to file bankruptcy?
    You need to reconstruct how much you don't have before you're allowed to file.
  • IT Girl (unregistered) in reply to dpm
    dpm:
    {many pointless guesses}

    Good heavens, am I the only one here who appreciates the clever show on the Disney Channel? As Doctor Heinz Doofenshmirz explains in almost every episode, the "Tri-State Area" is where Phineas & Ferb live.

    Seriously, it's a great show: the animation, the voices, but especially the writing. I heartily recommend it.

    <face plant> I can't belieive I missed that. It is a great show. Tri-state area is used because the Simpsons already stole the ubiquitous Springfield.

Leave a comment on “Death by Delete”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article