• (cs)

    Poor Dave. They probably started off as an informal shop and never grew up while the company (and app) became big.

    On the other hand, that reminds to verify if the backups I requested are actually happening. 8-O

  • (cs) in reply to John
    John:
    It always take an event like this for people to realize they *need* to test their backups every now and then... You're bound to have something like this happen if you don't.

    Yea we just recently had one of our customers get hit by this.

    It was a fairly small business so their server was actually just an XP workstation with no raid setup. Hard drive crash meant lost data. No big deal since they did backups every night that got sent to an external drive (and were proud to inform me of that when I asked). Except when I went to restore the most recent backup it was corrupt, as was the one before that, and before that, and before that, all the way back about 3 weeks to when someone had done a manual backup.

  • Bill's kid (unregistered) in reply to Jeff
    Jeff:
    "TRWTF is doing stuff like that interactively and auto-committing the transaction (or using a system without transactions in the first place). "

    I'm assuming the database is Oracle, given the use of Enterprise manager.

    DDL in Oracle auto-commits. No rollback.

    The Article:
    It was still running on SQL Server 2000"

    you know what happens when you ASS-U-me

  • Larry (unregistered) in reply to chron3

    Wow.

    With a bit of background, this would be a feature.

  • (cs) in reply to Bill's kid

    Well, in all fairness, Oracle does also have a product called Enterprise Manager.

  • nitehawk (unregistered)
    <!-- Babylon 5 was our last, best hope for peace. -->It failed. <!-- But in the year of the Shadow War, it became something greater: our last, best hope for victory. -->

    holy crap that was epic

  • Uhh (unregistered)

    Why on earth would he drop/recreate to deploy stored procedures? Someone should tell Dave about the "ALTER PROCEDURE" command.

  • Gram ma (unregistered) in reply to Jeff
    Jeff:
    I'm assuming the database is Oracle, given the use of Enterprise manager.

    You are? really? Even though it says SQL Server?

  • phleabo (unregistered) in reply to Jeff
    Jeff:
    I'm assuming the database is Oracle, given the use of Enterprise manager.

    I assumed SQL Server, given the use of Enterprise Manager and the screenshot of an SQL Server Enterprise Manager window accompanying the article, as well as the mention of the fact that the app "was still running on SQL Server 2000, but again, the upgrade was coming soon."

  • klystron_the_oven (unregistered) in reply to chron3

    That would be a real WTF moment.

  • Generic name (unregistered)

    Just checked on Enterprise Manager, and there's a checkbox to add the drop script before the create script for all objects. Had no one at the company actually looked at the Formatting tab?

  • Bob (unregistered)

    "he next day, John helped Dave's replacement, Phil, reverse engineer as much data as they could off of other servers and databases". Saw that coming :)

    I can picture that poor guy sitting there with beads of sweat popping up on his scalp...

  • (cs)

    Quality. I've worked in shops where they did promotions the same way - but under penalty of DEATH you offlined the database first, ran a backup, AND THEN started deleting shit. It's only 3 extra clicks and 2 more non-compiling excuses to do the "I'm waiting for it to compile" activities - one waiting for user disconnects before you force them, and one waiting for the backup to run.

  • arrowdrive (unregistered) in reply to Uhh

    There are some good reasons for that, especially on pre-SQL 2005. SQL only maintains a created on date, with dropping and creating you would always be able to tell easily which ones where recently changed. On my SQL 2005 and 2008 servers I have a DDL trigger that logs it all for me, so don't care all that much.

    And yes, my backups last night are good! ;) Restored every single night as soon as they are backed up. Excessive? Maybe, but better safe than sorry.

  • (cs)

    "Backup?" Dave said the word, but it held no immediate meaning. "Backup? Backup!" The semi sailed by, and Dave relaxed.

    This is possibly the most bizarre sentence I have ever read on DailyWTF. It's almost koan-like in its incoherence.

  • Alex (unregistered) in reply to Don
    Don:
    A few months later .. yeah it happened. Our systems died horribly. Seems _I_ was supposed to advise on ways around this, and hey, why didn't we have a Cold Storage that could have been fired up in this case..

    I think he still works there; and still reads 'Executive IT Management' magazines.

    This is why you should do these things with email trail.

  • My Name Is Missing (unregistered)

    FATAL: [Comment lost due to a database error]

  • Generic name (unregistered) in reply to arrowdrive

    But they're deleting and recreating every single proc, even those that haven't changed. The create date will provide no more tracability than altering them

  • (cs) in reply to Zylon

    People like you are the reason I always include an "Easy Reader Version" in the HTML comments. I can understand why you find it a difficult sentence to follow- it's several distinct sentences. I recommend paying attention to punctuation and grammar in the future. Those symbols actually convey some semantic information about the contents of the sentence and their relationship to the others. Also, this is one of those rare cases where one must remember what happened in previous paragraphs to properly comprehend what happens in this one. It's a tricky skill to master, but a valuable one to learn.

  • Bob (unregistered) in reply to hatterson

    I've seen this happen twice. The first time the sysadmin bumped a server into a door frame while he was carrying it - HDD crashed with no backup = insta-fired. The second time was a sysadmin who was asked to restore a few versions of an email database - this was a request made by police as part of a child porn distribution investigation. Backups done? Check. Backups safely stored offite? Check. Data on backup tapes? Oops. Insta-fire #2.

  • jrh (unregistered) in reply to Jeff
    Jeff:
    I'm assuming the database is Oracle, given the use of Enterprise manager.

    I'd assume it was MSSQL since it they were using SQL2000.

  • rob (unregistered)
    ...as he pulled the application up and logged into the test database

    TRWTF is that there was gigabytes of customer's live data in the test database.

  • Drew (unregistered)

    -Integer.MAX_VALUE for horrible horrible puns.

  • (cs) in reply to rob

    No, I think your reading comprehension is TRWTF.

    "With that done, he explained the next steps as he executed them: log onto production, navigate down the tree to where the stored procedures were, delete that entire branch of objects without mercy, and run the script to recreate them."

  • Anonymous (unregistered) in reply to SARUMANATEE
    SARUMANATEE:
    Nononono. Wait for it to finish then CTRL-Z. Cooler heads prevail.

    You can't send a task that is already finished to the background. :D (Linux joke for those not in the know. CTRL-Z on Linux pauses the currently running task and gives you a command line. Issuing the "bg" command from that command line restarts that task as a background process. CTRL-Z is also useful if you're running a time consuming process and want to quickly execute another command without having to spawn another terminal session.)

  • Clyde (unregistered) in reply to HAL

    Really like the I can haz data? animation!

  • (cs) in reply to Zylon
    Zylon:
    "Backup?" Dave said the word, but it held no immediate meaning. "Backup? Backup!" The semi sailed by, and Dave relaxed.

    This is possibly the most bizarre sentence I have ever read on DailyWTF. It's almost koan-like in its incoherence.

    I liked it. In context it made perfect sense to me. The use of imagery (or was the semi a metaphor?) reminded me of Stephen King's writing style for some reason. It actually made me miss reading novels, which I rarely do anymore.

  • (cs) in reply to ObiWayneKenobi
    ObiWayneKenobi:
    The thing I've noticed is that most of the time, there's no time or budget to HAVE a disaster recovery plan, so there's either a half-assed one put in place, or nothing at all.

    Again, management is the problem.

    We had some high priced consultants brought in by the CFO to do this stupid app. Nothing we couldn't have done in house, but, you know, people outside always seem smarter.

    They put it on our database server, and we're given strict instructions to NOT TOUCH IT, because our primitive little brains might break something. I even went out of my way to ask, "You sure you don't need me to do backups or anything?" Nope.

    So, predictably, the server craps itself. RAID card bites it, backup of the raid structure doesn't work, the array is corrupted beyond all recognition, etc, etc. I get the beep, I come in, I restore to the spare machine, everything comes up and is happy except the shmancy consultant package. I send them a message telling them to remote in and get their shit working again, and they tell me "We can't find the backup directory."

    ...

    I explain, "That machine is dead. Anything that was there, is gone."

    They say, "I thought it had a RAID?!?"

    I say, "That's what broke."

    They say, "..."

    And I got to go into the meeting and report that everything that I was responsible for, was recovered and working perfectly.

  • James (unregistered)

    Funny, as I read this I'm in the middle of writing up a request for a development environment. We're a 500M company and our testing and development is currently all done on production servers.

    It's fun living on the edge.

    I will say that we actually have a very robust backup system...a 10T SAN with all prev day backups on live spindles and every past week on a 30 casette tape drive. And yes, the backup system is "tested" in the sense that it's needed and used somewhat frequently!

    captcha jumentum: Israelis that keep moving

  • . (unregistered)

    OMG! Gigabytes of Ponies!

  • Some Wonk (unregistered) in reply to frits
    frits:
    Zylon:
    "Backup?" Dave said the word, but it held no immediate meaning. "Backup? Backup!" The semi sailed by, and Dave relaxed.

    This is possibly the most bizarre sentence I have ever read on DailyWTF. It's almost koan-like in its incoherence.

    I liked it. In context it made perfect sense to me. The use of imagery (or was the semi a metaphor?) reminded me of Stephen King's writing style for some reason. It actually made me miss reading novels, which I rarely do anymore.

    I read it more like the Blues Brothers. 'Elwood! The band!' 'The band?...The band.....The band!'

  • Ah yes (unregistered)

    Most of the developers I actually have respect for have, at one point in their career, deleted the database. It is a right of passage!

    Of course, when I did it I was on Oracle and had explicit transactions on so I was able to save my own bacon.

    Captcha, abbas as in dumb abbas

  • (cs) in reply to Ah yes
    Ah yes:
    Most of the developers I actually have respect for have, at one point in their career, deleted the database. It is a right of passage!

    Of course, when I did it I was on Oracle and had explicit transactions on so I was able to save my own bacon.

    Most of the developers I have respect for are embedded developers. I leave the finishing of the meme as an exercise for the reader.

  • History Teacher (unregistered) in reply to Guest
    Guest:
    And nobody thought, in the history of that company, of writing a simple script which, once working, would allways delete the right tree (maybe even do some sanity-checks before doing so) and automatically start the rebuild too ?
    I'm sure they did. That may even have been a regular coffee table discussion subject.

    Which is why the management should often hang around the peons during coffee breaks. Ok, that assumes a competent manager who can filter the critical pieces of information from the regular coffee table bullshit. So actually, there's no hope.

  • (cs) in reply to arrowdrive
    arrowdrive:
    There are some good reasons for that, especially on pre-SQL 2005. SQL only maintains a created on date, with dropping and creating you would always be able to tell easily which ones where recently changed. On my SQL 2005 and 2008 servers I have a DDL trigger that logs it all for me, so don't care all that much.

    And yes, my backups last night are good! ;) Restored every single night as soon as they are backed up. Excessive? Maybe, but better safe than sorry.

    If you need to look at the database to find its change history, then you have a source code control problem. Don't fix a source control problem by changing the deployment methodology, just source control the database schema.

  • luptatum (unregistered)

    If it was embedded system without file-system..(oh wait...)

  • logical.. (unregistered)

    The real wtf (read: made up) part of this story, is that within 24 hours, they fired the subject matter expert who would have been best placed to resolve the issue and interviewed and hired a replacement.

  • Ken B. (unregistered) in reply to Arancaytar
    Arancaytar:
    When deleting something takes even a tiny bit longer than you expect, that means you are deleting more than you expected. CTRL-C!
    Why would you want to copy your selection to the clipboard? :-) 
    rm -r *
    "What do you mean, I should run 'pwd' first?"
  • anoldhacker (unregistered) in reply to Don
    Don:
    ObiWayneKenobi:
    I would wager, sadly, that this is how the majority of "mission critical" systems operate. As always, the root cause is not the developer but incompetent management who have no idea of the right ways to do things.
    Well.. actually.. I find it's usually the management that ARE TOLD the right way to do things; but steadfastly deny that it's important, since "it never caused a problem before". Once upon a time I had a manager that loved reading 'Executive IT magazines' - you know, those ones written by people with MBA's and no IT knowledge whatsoever. Thick as a couple of planks when it came to IT, but it was the 90's and nobody cared so long as the manager could make Gantt charts. I had walked into his office the previous morning to discuss the budget for a disaster recovery cold site, since we were running 'mission critical' systems for our international business. 'No time today, come back tomorrow'

    So, with some quotes in hand and a few idea's on how we would keep data synchronized; I walked into the breach.

    'Sorry Donny, we'll have to pass on the DR Cold Site. Was reading an article last night about how it's a real white elephant, and we don't have the money right now for things that are not critical path'
    [Which critical path usually meant the CEO's new private jet]

    A few months later .. yeah it happened. Our systems died horribly. Seems I was supposed to advise on ways around this, and hey, why didn't we have a Cold Storage that could have been fired up in this case..

    I think he still works there; and still reads 'Executive IT Management' magazines.

    Rookie mistake: you should have sent him an email with the proposal. Archive the response immediately...

  • causa-causa (unregistered) in reply to logical..
    logical..:
    The real wtf (read: made up) part of this story, is that within 24 hours, they fired the subject matter expert who would have been best placed to resolve the issue and interviewed and hired a replacement.
    No TRWTF is that you didn't already figure out from the comments that Dave's replacement was someone else already on the team, who already knew the system, and could be trusted to do this stuff safely/correctly. How is it a WTF to fire the fuckup and give the task to a more competent programmer? On my team things are constantly getting reassigned to the more competent programmers -- if it wasn't for that, we would never get any projects finished! (Now if only the company had the balls to actually fire the fuckups......)
  • The Nerve (unregistered)

    I'll point out an interesting aspect of this WTF.

    1. The main problem was caused because a completely manual process failed.

    2. The failure to recover was caused because a completely automated process failed.

    I've been trying to find a balance between the two.

  • Eric (unregistered)

    Thats when you learn to generate the script that drop the stored proc for you.

    if (sproc exists) drop sproc create sproc

  • Uncle Al (unregistered) in reply to Some Wonk
    Some Wonk:
    frits:
    Zylon:
    "Backup?" Dave said the word, but it held no immediate meaning. "Backup? Backup!" The semi sailed by, and Dave relaxed.

    This is possibly the most bizarre sentence I have ever read on DailyWTF. It's almost koan-like in its incoherence.

    I liked it. In context it made perfect sense to me. The use of imagery (or was the semi a metaphor?) reminded me of Stephen King's writing style for some reason. It actually made me miss reading novels, which I rarely do anymore.

    I read it more like the Blues Brothers. 'Elwood! The band!' 'The band?...The band.....The band!'

    For old NFL fans like myself, it was also reminiscent of Jim Mora responding to a reporter's question about his team making the playoffs.

  • (cs) in reply to satancom
    satancom:
    Thats a classic, its like leaving the 'where' statement out of an update querey.

    I've done that. We use Livelink 9.7.1 here. Sometimes the reservation information for a document will go kablooey, so it can't be unreserved in the app. Direct table manipulation of three columns is needed.

    Once a misplaced semicolon meant that I lost the reservation information for about 950 documents.

    I performed a rollback, and grabbed another sip of Pepsi, confident in myself, and thinking "that's why you always start with BEGIN"

  • JSR (unregistered)

    DBA here. I don't get this whole "no backup for months" thing that I keep seeing in different WTFs.

    Every place I've ever worked, we're constantly cloning the production database down over the development/test systems. And this is always done using the normal production backups as a source. If those backups don't work/are out of date, you find out pretty quickly when you can't build the new Dev/Test.

  • Plz Send the Code (unregistered)

    Doesn't a box come up that says something like "You're about to delete all the tables. Are you sure you want to do that?"

  • wtf (unregistered) in reply to Plz Send the Code
    Plz Send the Code:
    Doesn't a box come up that says something like "You're about to delete all the tables. Are you sure you want to do that?"

    No, it's a little animated nuclear bomb. It says "It looks like you're trying to nuke your database! I've just done that for you, are there any backups that I can delete while I'm at it?"

    I hate Clippy, but Kablooey is just evil.

  • one space to many (unregistered)

    I remeber a friend of mine doing something similar:

    someone@some-computer:~/aVeryImportenetDir# rm -r oldFilesToDelete/ *

    But where did that space come in, that shouldn't be there!

    it was hilarious and also the backup server was down so it made it that much more funny. Lucky for him, all that got lost was data that could be recovered with a lot of work :)

  • Mike (unregistered) in reply to Jaime
    Jaime:
    Where I work, our deployment processes are begging for this to happen. The previous manager demanded that all deployment steps be executed by the user, scripting or automating the steps was considered obfuscation and not allowed. So, we regularly had deployments that involved copying hundreds of artifacts to productions systems and manipulating them by hand. Only very recently have I been able to convince management that an escrowed script is way more reliable and testable than a piece of paper.

    BTW, the previous manager's justification for no scripts was that the developer could change a script after testing, but the deployment plan with individual steps could be printed out and therefore made unchangable. Apparently the manager had not heard of file-system security.

    Or white-out.

  • JB (unregistered)

    TRWTF is that Dave was replaced by the company. He should have had a backup plan like blaming John for the cock up.

Leave a comment on “Production Promote”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article