• SnapShot (unregistered)

    I've done something similarly stupid.

    Boss: "Hey, SS, go and delete some values but keep the important rows in the (production) database."

    Me: 

    delete from critical_table where foo <> 'important;
    Me: Hmmmm... why is this not running? 
    randomly hitting keys (and appending to my where clause)
    Me: Oh, forgot to close the quote. 
    hits enter
    Me: Oh shit... (luckily our backups where daily)

  • (cs) in reply to Bill's kid
    Bill's kid:
    wtf:
    Darth Database:

    I have backed up the database. Pray I don't back it up any further.

    Keep trying. One day it'll be funny, really.

    Give him a break, it's his "joke" version of a hammer so everything looks like a comment version of a nail.

    If he backs it up any further, he'll crush the rear bumper and brake lights against that parked truck there!

    beeeep ..... beeeep ..... beeeep ..... danger! ..... database is reversing .... beeeep ..... beeeep ..... beeeep ....
  • Matt Westwood (unregistered) in reply to Ah yes
    Ah yes:
    Most of the developers I actually have respect for have, at one point in their career, deleted the database. It is a right of passage!

    Of course, when I did it I was on Oracle and had explicit transactions on so I was able to save my own bacon.

    Captcha, abbas as in dumb abbas

    Yep, I did it once, but it was only a small one and I managed to put it back together before anyone noticed. I was sitting in a puddle of perspiration after I'd finished. At least, I think it was only perspiration.

    CAPTCHA: letatio: an unnatural act with a vegetable?

  • robbo (unregistered)

    We apologise for the database llama outage. The people responsible for firing the people responsible, have been fired.

  • Franz Kafka (unregistered) in reply to GalacticCowboy
    GalacticCowboy:
    Actual event at a former company: DBA was dinking around in the production database. 
    update UserTable
set EmailAddress = '[email protected]'
--where UserId = 347

__________________

(15347 rows updated)

    "... That doesn't seem right."

    /headdesk

    Fortunately, we had enough internal controls in place that he was able to recover everything within a couple of hours.

    How long does it take to type 'rollback;'?

  • (cs) in reply to Franz Kafka
    Franz Kafka:
    GalacticCowboy:
    Actual event at a former company: DBA was dinking around in the production database. 
    update UserTable
set EmailAddress = '[email protected]'
--where UserId = 347

__________________

(15347 rows updated)

    "... That doesn't seem right."

    /headdesk

    Fortunately, we had enough internal controls in place that he was able to recover everything within a couple of hours.

    How long does it take to type 'rollback;'?

    He probably doesn't have the option because it's Microsoft SQL Server, which doesn't automatically create transactions by default. You can only type "rollback" if you have previously typed "begin transaction". It looks like MSSQL because the trailing semicolon is missing (it's optional in MSSQL).

  • nerfer (unregistered) in reply to Don
    Don:
    ObiWayneKenobi:
    I would wager, sadly, that this is how the majority of "mission critical" systems operate. As always, the root cause is not the developer but incompetent management who have no idea of the right ways to do things.
    Well.. actually.. I find it's usually the management that ARE TOLD the right way to do things; but steadfastly deny that it's important, since "it never caused a problem before". Once upon a time I had a manager that loved reading 'Executive IT magazines' - you know, those ones written by people with MBA's and no IT knowledge whatsoever. : : So, with some quotes in hand and a few idea's on how we would keep data synchronized; I walked into the breach.

    'Sorry Donny, we'll have to pass on the DR Cold Site. Was reading an article last night about how it's a real white elephant, and we don't have the money right now for things that are not critical path'

    A few months later .. yeah it happened. Our systems died horribly. Seems I was supposed to advise on ways around this, and hey, why didn't we have a Cold Storage that could have been fired up in this case..

    I think he still works there; and still reads 'Executive IT Management' magazines.

    That's why I do things in e-mail as much as possible. Then when somebody contradicts themselves a few months later, I dig up the old e-mail and remind them what actually had transpired. Even if I talk to somebody in person or attend a meeting, if there's some important outcomes from that, I'll send an e-mail to interested parties saying "Let me verify that I understand this correctly: ..."

  • Franz Kafka (unregistered) in reply to Jaime
    Jaime:
    Franz Kafka:
    GalacticCowboy:
    <oh shit moment>

    How long does it take to type 'rollback;'?

    He probably doesn't have the option because it's Microsoft SQL Server, which doesn't automatically create transactions by default. You can only type "rollback" if you have previously typed "begin transaction". It looks like MSSQL because the trailing semicolon is missing (it's optional in MSSQL).

    Part of any deployment plan involving SQL is some variant of 'set autocommit off', at least when I've done them. partying on production data without a safety net makes my blood run cold.

  • (cs) in reply to Franz Kafka
    Franz Kafka:
    Jaime:
    Franz Kafka:
    GalacticCowboy:
    <oh shit moment>

    How long does it take to type 'rollback;'?

    He probably doesn't have the option because it's Microsoft SQL Server, which doesn't automatically create transactions by default. You can only type "rollback" if you have previously typed "begin transaction". It looks like MSSQL because the trailing semicolon is missing (it's optional in MSSQL).
    Part of any deployment plan involving SQL is some variant of 'set autocommit off', at least when I've done them. partying on production data without a safety net makes my blood run cold.
    The "oh shit moment" you were quoting included the phrase "DBA was dinking around in the production database". I'm pretty sure this implies that he didn't have the foresight to turn off autocommit.

    BTW, why bother turning autocommit off in a real deployment plan? This seems to imply that the DBA will be tinkering. My deployments simply use begin, rollback, and commit as necessary. I don't create plans that have a point where a human can intervene and roll a single transaction back. I'm sure someone out there is thinking that you might leave a transaction uncommitted and go verify that the application works, but it is highly unlikely that you can verify the data that has been modified (and is still locked) by a transaction from another session.

  • PRMan (unregistered) in reply to Darknite
    Darknite:
    We have all made screw ups, it seems there was not proper Diaster Recovery in place, and I actually feel sorry for Dave.

    Of course standard practive is always make a backup before doing a push to live.

  • Stiggy (unregistered)

    TRWTF is a working environment where developers have any more than SELECT access to production.

    It was an accident waiting to happen.

  • jon (unregistered)

    I really feel for this guy.

  • Häl (unregistered)

    I'm afraid i shouldn't have let you done that Dave. Dave, stop. Dave, stop.

  • Franz Kafka (unregistered) in reply to Jaime
    Jaime:
    BTW, why bother turning autocommit off in a real deployment plan? This seems to imply that the DBA will be tinkering. My deployments simply use begin, rollback, and commit as necessary. I don't create plans that have a point where a human can intervene and roll a single transaction back. I'm sure someone out there is thinking that you might leave a transaction uncommitted and go verify that the application works, but it is highly unlikely that you can verify the data that has been modified (and is still locked) by a transaction from another session.

    Simple - the deployments I've done have been largely incremental, and turning off autocommit allows some sanity checking for updates and the like. Generally, I'm updating a known amount of metadata, so I expect perhaps 5 rows to be affected - if I get 100, I roll back and figure out why.

  • (cs) in reply to Franz Kafka
    Franz Kafka:
    Jaime:
    BTW, why bother turning autocommit off in a real deployment plan? This seems to imply that the DBA will be tinkering. My deployments simply use begin, rollback, and commit as necessary. I don't create plans that have a point where a human can intervene and roll a single transaction back. I'm sure someone out there is thinking that you might leave a transaction uncommitted and go verify that the application works, but it is highly unlikely that you can verify the data that has been modified (and is still locked) by a transaction from another session.

    Simple - the deployments I've done have been largely incremental, and turning off autocommit allows some sanity checking for updates and the like. Generally, I'm updating a known amount of metadata, so I expect perhaps 5 rows to be affected - if I get 100, I roll back and figure out why.

    Interesting. When my updates get installed, the person implementing them treats it like a commercial software installation. I've never looked in the Program Files directory to verify that Microsoft Office installed properly. Of course, a production deployment is never new territory for us, we have already installed it in Test, QA, and Stage before production. If there are any doubts as to whether it will work, we test it again. Due to change control timelines, a failed deployment can't be retried for at least two weeks, so we spend a lot of time making sure deployments never fail.

  • (cs) in reply to Ben4jammin
    Ben4jammin:
    I am always amazed at these "no backup" stories. At our shop, I have to do monthly test restores and document it. I have to make sure to do test restores of files and DBs, both local B2D and off-site B2D. AND it is the first thing discussed in my yearly reviews. In a related story, we have avoided more than one WTF because we have good backups. Like when the VP of IT (former SQL guy) forgot the WHERE clause in his delete statement and wiped a whole table (apparently a very important one) late at night. Next morning, no one was the wiser except him, me, and our services manager. And since he does our reviews, we aren't telling :)

    Damn, you are the kind of person who kills thedailywtf because no new story will even created with that kind of system in place.

  • Ben (unregistered) in reply to Remy Porter
    Remy Porter:
    Grunt:
    TRWTF is doing stuff like that interactively and auto-committing the transaction (or using a system without transactions in the first place).

    Standard SQL- DDL statements never happen in a transaction. "DROP TABLE" can't be rolled back.

    ANSI / ISO, as of 2006, says: "It is implementation-defined whether or not the execution of an SQL-data statement is permitted to occur within the same SQL-transaction as the execution of an SQL-schema statement." (4.35.1)

    Some DBMS's certainly can, notably PostgreSQL. I believe Sybase always could and SQL Server is essentially derived from Sybase. Others have various ways of saving you, e.g. Oracle autocommits DDL statements, but has a "recycle bin" to recover inadvertently dropped tables and has a feature that can generally guarantee "undoing" actions within a certain amount of time without taking the server offline.

  • (cs) in reply to No one
    No one:
    I have to admit though that most places I've worked were seconds from disaster on any given day. The procedures for making changes in the db were ad hoc an certainly a misplaced delete could spell disaster. As far as backups I've seen long periods of neglect while the organization put all it's effort into just growing the features and keeping the lights on. Even if the backups had been decent this example would still have been a large outage and they would have likely lost several customers anyway. Sometimes it just comes down to luck.

    You need a new keyboard. Apparently the comma key on yours is broken.

  • Another Dave (unregistered)

    I was going to share this story with my colleagues yesterday morning, with a cheery note saying "At least we've never had a disaster like this one!"

    Then I discovered that our team leader had wiped out a large chunk of irreplaceable customer data from one of our database the night before.

    Unlike the hapless Dave in the story, our team leader won't lose his job, but I think I'll save this gem for another day!

  • (cs)

    And that is why you need to usually keep devs away from production systems.

    And of course you need to separate development and production by an organizational firewall in the first place.

  • (cs) in reply to Coyne
    Coyne:
    Guest:
    And nobody thought, in the history of that company, of writing a simple script which, once working, would allways delete the right tree (maybe even do some sanity-checks before doing so) and automatically start the rebuild too ?

    Nowadays, there seems to be a general aversion to expenditures of time for internal automation. The theory seems to be that it is better to spend 3 hours each time to do some manual process than to spend 30 hours to build a tool; even though the result will be thousands of hours expended on the manual process over a year and the 30 hours is expended once.

    "The cobbler's children have no shoes." We're so busy automating the world we have no time to automate our own work.

    One of my pet peeves.

    It is being so busy running that you don't have time to stop in order to get into the car ...

    I usually charged customers for internal development - they needed the same scripts and hey, presto! Everybody was happy.

  • (cs) in reply to Coyne
    Coyne:
    Guest:
    And nobody thought, in the history of that company, of writing a simple script which, once working, would allways delete the right tree (maybe even do some sanity-checks before doing so) and automatically start the rebuild too ?

    Nowadays, there seems to be a general aversion to expenditures of time for internal automation. The theory seems to be that it is better to spend 3 hours each time to do some manual process than to spend 30 hours to build a tool; even though the result will be thousands of hours expended on the manual process over a year and the 30 hours is expended once.

    "The cobbler's children have no shoes." We're so busy automating the world we have no time to automate our own work.

    One of my pet peeves.

    It is being so busy running that you don't have time to stop in order to get into the car ...

    I usually charged customers for internal development - they needed the same scripts and hey, presto! Everybody was happy.

  • Nagi (unregistered)

    Poor Dave.

    Early on in my career, an old hand explained to me how to use the DOS delete command. He said "When ever you have a command like this on the screen an alarm bell should go off in your head. Check and double check before you hit return". It was very sound advice.

  • anon (unregistered)

    TRWTF.. (forgive me if this was already brought up as I haven't read all the comments) is why the f(&* was that bonehead Dave touching a production sql server???? Any organization that has "mission critical" data in production should always have a separate dev environment to develop against, build test install scripts, wipe out stored procs or whatever. That way when people like Dave do their fat finger stunts they can still create a new instance of the database in the DEV environment without screwing up production. Heck, I don't even fully blame Dave for this... Much of the blame belongs to the managers of this organization for not making sure a DEV SQL Server was in place.

  • Dan (unregistered)
    It was a small team, with the stereotypical alpha-geek at the top, people like Dave in the middle

    Maybe it should have been someone like Kevin.

  • Baktru (unregistered)

    Create Or Replace on Stored Procedures anyone?

  • another user (unregistered)

    Say, this company wasn't by any chance providing a software for the logistics industry?

    I'm just asking because I think I was working at the same place at that time ;-)

  • Richard (unregistered) in reply to Don
    Don:
    Well.. actually.. I find it's usually the management that ARE TOLD the right way to do things; but steadfastly deny that it's important, since "it never caused a problem before".

    I can't help but wonder how much of that is true because most IT people have very little business experience? If the business folk and IT folk actually work together to determine both the price of failover (high) and the cost of downtime (also high), they can figure out which decision makes sense for the company and both feel good about it. But no, its easier to point fingers...

    Most IT cost is an investment. Many investments are reasonable, many are not.

  • Anonymous (unregistered) in reply to Nagi
    Nagi:
    Poor Dave.

    Early on in my career, an old hand explained to me how to use the DOS delete command. He said "When ever you have a command like this on the screen an alarm bell should go off in your head. Check and double check before you hit return". It was very sound advice.

    Do you really need an old hand to explain this to you? Isn't common sense enough? I've been in this industry for over a decade, I've never been taught to "check and double check" a DOS/UNIX command or a critical SQL command. I just do it because I'm not a complete moron and it's pefectly obvious you need to treat irreversible commands with care. I can't be the only one who has never deleted critical company data through sheer stupidity, surely?

  • erewhon (unregistered)

    The ultimate 'OHNO second'!

  • The trouble with GUIs... (unregistered)

    is that errors like this one are all too easy. Dave was the victim of a ridiculous process. Never mind the 17 ways this error should have been prevented, it's much harder to cause a disaster when you are typing "@install_stored_procedures.sql"

  • Shinobu (unregistered)

    I must admit that I feel kind of sorry for Dave. It wasn't just his fault. Part of the problem was that no one was checking up on Dave. If you delegate some of your responsibilities to your underlings, you also put a process in place, so that when things start to slip, you know it before disaster strikes.

  • AK (unregistered)

    One of our production db is about 800 gigs! Hope i don't do something like that ;)

  • Pup (unregistered)

    How did he log into the test environment and delete production environment data?

    Sounds fishy.

Leave a comment on “Production Promote”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article