- Feature Articles
- CodeSOD
- Error'd
-
Forums
-
Other Articles
- Random Article
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
Admin
It's kind of suprising that the employee fixing the database wouldn't make a backup or at least use transactions before just deleting.
Oh well, many many people to blame in this tragedy.
http://dealpi.ejoop.com
Admin
Admin
TL;DR some guy wiped a database and the company didn't have backups, went bankrupt.
Admin
maybe, but depends on how much the 'right' plan would have costed. Lets say, theyre not morons, and they go to a plan for say, 200 a month. they've already spent 20 times that, that's over a year and a half i'd make if they weren't stupid. and i now have the chance to sell the capacity again.
Admin
At my first job (back in 1989), I worked with a certain language/technology that seems to be named after a disease (but is, in fact, named after the hospital where it was invented) and has appeared in several articles and sidebars here. One of the developers there (not me) wrote a command-line utility to delete data from the (non-relational) database. When you ran it, you got a prompt, and you typed in the reference to the data to delete. Being a developer, he wrote standard wildcard capablity into it.
The applications this company sold ran on VT-style terminals, and used a curses-like library to put forms on the screen for data entry. The fields had conventions for field input, including the following:
If you entered ^ into a field, you were moved to the previous field.
If you entered * into a field, you exited the form, cancelling all changes.
One day, a user was poking around the command line and found the data deletion tool (you thought I'd gotten off-track, didn't you!). He ran it, saw the "DELETE" prompt, and did what users of this company's applications did to cancel a form - he entered *.
Remember up above that I said the developer put wildcard support into the deletion tool?
Luckily, the customer did have backups ...
Admin
I worked for a company for 8 years that did a manual tape backup every night. They put the tape in before they left, and removed it the next morning. They used Microsoft Backup to do the actual backup. The tapes were even rotated so they could go back a week if they needed to recover any files. I never recall anyone actually doing a restore operation however, as the loses were normally considered too minor to be worth the time searching for the files.
When I left some 8 years later I noticed they were still using the same set of tapes, which happened to be a pack of 10, with each day of the week written on the first 7 (yes, they even backed up at weekends when staff levels were about 10%), and 3 unused and still wrapped up.
I do wonder at times whether the company will suffer the same fate at some in the future. I've not seen them in the news yet.
Admin
OMgosh dude that is just too funny!
RT www.anon-tools.tk
Admin
Admin
Perhaps you are correct that in the Idaho panhandle it is referred to in this way (though no one I know from Idaho has ever referred to it thus), but in Oregon and Washington, it is not known as the Tri-State area at all. This is for 2 reasons, I assume: 1) the actual point is under water - the snake river, and 2) not a whole lot of people live around there. In fact, one might very well consider it desolate.
Also consider, just because a business decides to take a name like "Tri-State" does not automatically mean that the term is in common usage.
Admin
Well I think those mistakes are hust too big even for management.
It was a scheme decison makers had to stole mone from the shareholders.
Admin
[quote]Who knows, growing up NY/NJ/E. PA was the tri-state area, and I was in my own little world, so it didn't dawn on me until I moved across the state that it could refer to more than one area.{/quote]
When you actually count them out, there are around 60 places where three states come together or come close enough to one another to be called a "tri-state area". Granted, places like the Oregon-Idaho-Nevada tri-state area aren't going to come to many people's attention, but there are DOZENS that will, mostly back east.
Admin
When exactly was this? If it's any time even remotely recently then Rick's ISP was a huge ripoff. $75 a month for shared hosting? For like $35 per month I have a managed VPS (virtual private server) with like 30GB hard disk space (of which I can use as much as I like for databases), 400GB bandwidth and they back it up every other day. This is with a relatively expensive VPS provider.
Hell, on shared hosting you can probably get a better deal than what MegaPetCo got for like $10 a month.
It's kind of funny to see people saying stuff like "for only $75 a month why would you expect to get backups?" when you can get fully managed service including backups for far less.
Admin
Ooh, a Monty Python reference!
Allow me to pile on by saying good riddance to incompetent idiots. The only two bad things are:
The poor developer who ran the query. He would have shit his pants 10 times over, and felt like crap for ages. He missed out on a resume reference too.
The employees who got laid off. The fact that a completely incomptent data infrastructure, put in place by completely incompetent morons, cost so many people their jobs ought to be grounds for some sort of compensation or punishment. These people deserve never to be in charge of anything with any sort of responsibility ever again.
Admin
probably because no one lives in the corner, what with its being a desert and all. tri-state areas get called that when there's a lot of state-line crossing going on in daily life.
Admin
LOL if only I could kill companies with a single SQL delete..
Admin
New South Wales, Victoria, South Australia, or New South Wales, Queensland, South Australia?? The closest Tri-State Area round these parts....
Admin
What about Western Australia, South Australia & Northern Territory?
Although, Northern Territory technically isn't a state...
Admin
Older versions of MySQL don't have transactional features !
But the real WTF is that if the client was paying about 50 x the base service, then the ISP could have at least thrown in better hosting and backups for that sort of money.
I often have cheapstake clients who through false economy pay more than they need to. Rather than gouging them, I provide extras without telling them about it. Then when disaster strikes, I can "do magic", charge reasonably for the effort to restore and win the day.
Another WTF is the ignoramus management that will allow some twerp to implement such a brain-dead system.
Admin
Thank you, on behalf of those of us too embarrassed to ask. I just looked it up on Google Maps, and I never realized Chicago was so close to two other states' borders...
Admin
REALLY? The SINGLE biggest WTF is that fact that everything was not only stored in one database, it was stored in a single table.
Sure, the WTFs abound. No Backups. BUT if things were in separate tables (and separate databases) the fact there was no backups might still not have been discovered. Some of the WTFs happened BECAUSE it was in a single table.
And of course we can forget the WTF of, "No Really we want to pay this, because we think you are trying to pull the wool over our eyes and make us pay more... by offering it for free."
Admin
Hilariously, it also proved incredibly expensive.
Admin
Yes but they knew how much money they didn't have since their database was still running.
Admin
Sad for the employees, but as others have said, this company deserved to fail. What person could be so moronic as to end up in that situation with their business? No IT staff? Not even a freaking part-time intern or outside help from a local contractor to setup a decent operation? It's just psychotic that ANY business would operate like that...
Admin
Well I just hope some of the columns were nullables. I just can't imagine the "insert" if not...
Admin
and so did Dara Ó Briain... :)
Admin
Not sure how it works in other parts of the world, but here, in the Netherlands, not having backups when something goes wrong you can be held responsible for negligance.
Admin
No, this is why you NEVER EVER WRITE A DELETE QUERY.
Seriously, if you have an application that uses a database, as opposed to "I store everything in the database and write everything, SELECT, INSERT, UPDATE, DELETE, manually. I really love how efficient the mysql command-line client is for running a business!", there is no reason you should ever be running a DELETE right in the database.
Your application should have a delete_item() method which calls a delete_row() method which calls the actually DELETE, and all of that should generate in such a way that a WHERE-clause is always present.
Special bonus: SQL has shitty syntax anyway, so you now avoid that.
Admin
Admin
Business has a poorly designed database. Developer inadvertently deletes all records. There are no backups. Business has to shut down.
There. I see only two WTFs here, and those are "daily" indeed. Of course, one should be careful of putting both in the same basket...
Admin
[quote user="dpm"]I would expect a real business to backup its own data, apparently that wasn't the case here either. Along with a few other cases. Like not hosting your company's entire IT infrastructure on a remote server.[/quote]I'm thinking of what happens if I screw up (hard drive crashes, I mistakenly wipe out a filesystem, or even just one file gets deleted by accident). If I cause the problem, I would think that I would be liable to some degree; therefore, I make backups of everything for my sake.[/quote]
You can bet the fella from the story does that now, assuming he's not asking if you want fries with that.
Admin
Lol!
With luck the guys at Megapet will be more careful in their next jobs and pay attention to their IT infrastructure.
A good ending story ; some place left in the ecosystem for better business to settle in.
Admin
Admin
The company was apparently effectively paying around $4k/month. I'm sure the ISP would not have switched them to a dedicated server if they had continued to pay $75.
Now regardless how stupid the company was - this probably made them one of the ISP's best customers, right? Now if you have such a good customer and you realize they are technically incompetent - why not setup a backup for them anyway? Them going broke doesn't just lose hundred's of employees their jobs, it also means the ISP lost a nice stream of revenue. So really - in this story there are two companies which should have had better business sense.
Admin
Tri-State has to be TRUE, FALSE, FILENOTFOUND. That's the only thing that makes sense.
The ISP lost a huge cash cow here. The managers of PetCo were (obviously) technically incompetent, but that sort of people tend to appreciate bullshit when ladled up.
Here's what should have happened when the slowness reason was discovered:
Admin
http://www.geocities.jp/hitotsubishi/w95_fdisk_format/
But yeah, sure, no one else had such troubles because no one other than me was silly enough to try examining their backups after making them.
Let me guess. You worked at Microsoft for a while, but no more ... maybe were you fired for reporting what happens to backups?
Admin
man, what do you eat? I could probably list every item I've ever eaten in 10k! It's not a grocery list has multiple entries for the same item, you just store it as key/value pair. Even using XML shouldn't take you beyond 1MB (and not even I use XML for my grocery list!)
Admin
So you do a begin; delete ...; rollback;
And then you realize that the database is mysql with MyISAM so you have still lost all your data.
They used a cheep shared hosting service, so it's a fair guess that they also used a mysql with default settings.
Admin
Certainly, backing up by itself is not required - one could have their books on paper. But if financial records are kept on computer (without paper copies) one must have at least two independent copies. For example a simple RAID setup would not suffice, as even there are two physical copies, they're not independent and would not survive an accidental delete command.
Admin
It's a nice assumption to assume such laws exist. Years ago I came into an accounting with firm who'd just had a server crash. Their 'backup' involved inserting a floppy disk into the server drive every night. Noone had ever realised that backups actually need some kind of setup, or that their 40GB database wouldn't ever fit onto a floppy disk. After much effort, we recovered enough of the database onto a repaired server (with only a few MB gone AWOL in the repair..) that they were operating again. They asked us to take over managing IT, which we did, and naturally the first recommendation we made was for a backup system. They made all sorts of silly purchases, such as a gigabit switches (while refusing to upgrade the server NIC from 100M) and gaming video card installations, but management had come down with their firm view: we had recovered the server once, so we could do it again. We pushed the point that we couldn't guarantee that, but the response was "if you won't do it, we'll get someone who will". Let me remind you we're talking about a company that processes other businesses financial data. Then they got audited, and went through SOX compliance. This is about as tight as it gets in IT. SOX compliance works by paying for an audit and "identifying and managing risk". So when the auditor said "you have no backups", management said "yes, and we're OK with that". Bang, fully compliant company.
If a company can get through a signed audit by a much bigger and well known accounting firm on the basis that they know what they are in for when that server fails, well that says something about there being no law on that subject.
I've since been involved with a hosting environment. Nearly every customer was happy to acknowledge a complete lack of backups if it meant $5 got shaved off their monthly charge.
It's easy to get on a high horse about people being fired over lack of backups, but there's usually someone in accounts who outranks them who made that decision.
Admin
I do realize however, that even when a law exists, not everyone chooses to abide it. And probably most companies aren't even aware of every law affecting them.
Admin
Should be "NSW/QLD/SA" since if we are supposed to decipher "NY/NJ/CT" they can decipher our state codes too! Though in a country with only six states and requiring 3-letter state codes it a minor WTF...
Admin
Admin
Yeah but unlike these guys their financial systems were running to the end. With Enron, I'm fairly certain the real numbers were known by someone at the place. The inmates may have been running that asylum but there had to be one figure of sanity somewhere...
Admin
You don't need to look at the contents of a file to see how big it is. Same goes for databases.
Admin
Admin
Buy the cheapest pc/disk/router you can get away with. Oh we need a firewall? Get the cheapest and leave the default permissions/rules on the firewall b/c it works. Wireless hookups? It's too expensive to call someone every time we need to configure a new machine; leave the defaults. The PC disks last 5+ years, and we're in business 3 months; we don't need backups b/c we have time to look at it later...
I'm surprised they didn't follow the patterns of the one thing Wall Street actually got right! A few years back there was a massive power outage that took out most of Manhattan, and getting a cab to our backup site in NJ wasn't practical in the timeframe available. Buildings went dark, and a few of the main UPS units blew out so everything was off line. But the phones still worked. They just went back to doing everything with paper order slips and let an army of clerks sort it out after the fact. And these are exceedingly high volume businesses. I can't imagine a pet supply store had that much transaction volume...
Admin
WTF was the primary key?
CAPTCHA: amet - what many men need to do better
Admin
You expect a primary key in this mess? What for? The customers, bank accounts, products, transactions or some other business entity? Oh wait, perhaps they opted for a UUID...
Admin
Ouch. So that's what happens when that question is answered the other way. Of course, I've heard rumors that the results of answering things that way is that any actual financial data loss can result in jail time for the management individuals who said they were OK with it...