• Anon (unregistered)

    101st!

  • Leroy Jenkins (unregistered)

    John: is everyone clear on the stored procedure replacement plan? John: Dave? ... where's Dave? ... Dave: Alright I'm back! Let's do it! John: no no wait! Dave! The plan! Dave: LEEEEEROOOOOOY JEEEEENKIIIINSS!!!!

  • Bosshog (unregistered) in reply to anon
    anon:
    You mean the one everyone else has noticed ad nauseum?
    Maybe he is a f a s t e r  r  e  a  d  e  r   t   h   a   n   y    o    u.
  • Dev talk (unregistered) in reply to JSR
    JSR:
    DBA here.
    Shh! he's here. pretend we weren't just talking about him..
  • I Forgot My (unregistered) in reply to Remy Porter
    Remy Porter:
    People like you are the reason I always include an "Easy Reader Version" in the HTML comments. I can understand why you find it a difficult sentence to follow- it's several distinct sentences. I recommend paying attention to punctuation and grammar in the future. Those symbols actually convey some semantic information about the contents of the sentence and their relationship to the others. Also, this is one of those rare cases where one must remember what happened in previous paragraphs to properly comprehend what happens in this one. It's a tricky skill to master, but a valuable one to learn.
    What is a valuable one to learn? You lost me there.
  • George-Michael (unregistered) in reply to Sam
    Sam:
    Haha, from the source code: <!-- Dave whipped around in the chair so quickly that he sliced his arm off on the edge of a server rack. He screamed as blood pumped from the stump; John screamed. Suddenly, Dave stopped screaming, the fake blood stopped pumping, he grabbed John by the collar, and said, "And *that's why you have a robust install process for production!*" -->

    Also

    <!-- And that's why you don't teach people lessons! Sorry for the pop-culture extravaganza. Okay, not really. --><!-- Easy Reader Version: See Dave. See Dave perform without a net. See Dave break his neck. See Phil get a promotion. -->

    TRWTF is who is Phil?!

    TRWTF is Arrested Development ever being canceled.

  • Anon (unregistered) in reply to Remy Porter
    Remy Porter:
    No, I think your reading comprehension is TRWTF.

    "With that done, he explained the next steps as he executed them: log onto production, navigate down the tree to where the stored procedures were, delete that entire branch of objects without mercy, and run the script to recreate them."

    No, your comprehension of "reply" vs "quote" is TRWTF.

  • (cs) in reply to The Nerve
    The Nerve:
    As bad as this sounds, nothing beats the thrill of performing without a net.
    I prefer performing without .NET.
  • Man Ager (unregistered) in reply to ObiWayneKenobi
    ObiWayneKenobi:
    As always, the root cause is not the developer but incompetent management who have no idea of the right ways to do things.
    So, I can spend $BIG_BUCKS on disaster recovery stuff for something that will probably never happen, or roll the dice, save the company tons of money, and get a big fat bonus. Given that I have a rock-solid opinion of my own infallibility, what am I going to do?

    That's OK, take your time to puzzle it out. I'll wait.

    That's right, I spend the $BIG_BUCKS on yet another report generator sold by my cousin's basement software company in which I hold a 30% stake.

    Rookies!

  • wtf (unregistered) in reply to Man Ager
    Man Ager:
    ObiWayneKenobi:
    As always, the root cause is not the developer but incompetent management who have no idea of the right ways to do things.
    So, I can spend $BIG_BUCKS on disaster recovery stuff for something that will probably never happen, or roll the dice, save the company tons of money, and get a big fat bonus. Given that I have a rock-solid opinion of my own infallibility, what am I going to do?

    That's OK, take your time to puzzle it out. I'll wait.

    That's right, I spend the $BIG_BUCKS on yet another report generator sold by my cousin's basement software company in which I hold a 30% stake.

    Rookies!

    Not hard a hard call, since you know there will always be some developer around to blame for the lack of business continuity planning.

  • Bob p (unregistered)

    Years ago I remember going into a shop to borrow their tape drive. It didn't work but there was no indication that they knew, until I looked in the log book and found several months of entrees reading "backup failed". Cue hilarious round of blame storming. The problem was a stuck bit in the data interface.

  • Bob p (unregistered)

    Years ago I remember going into a shop to borrow their tape drive. It didn't work but there was no indication that they knew, until I looked in the log book and found several months of entrees reading "backup failed". Cue hilarious round of blame storming. The problem was a stuck bit in the data interface.

  • jugis (unregistered) in reply to Bob p
    Bob p:
    Years ago I remember going into a shop to borrow their tape drive. It didn't work but there was no indication that they knew, until I looked in the log book and found several months of entrees reading "backup failed". Cue hilarious round of blame storming. The problem was a stuck bit in the data interface.
    They must've been really full after eating all those entrees. I guess it was copying from a FAT drive not a thin client?
  • quisling (unregistered) in reply to John
    John:
    It always take an event like this for people to realize they *need* to test their backups every now and then... You're bound to have something like this happen if you don't.
    If you don't restore your backup as regularly as you take it, you don't HAVE a backup.
  • Doozerboy (unregistered) in reply to Leroy Jenkins
    Leroy Jenkins:
    John: is everyone clear on the stored procedure replacement plan? John: Dave? ... where's Dave? ... Dave: Alright I'm back! Let's do it! John: no no wait! Dave! The plan! Dave: LEEEEEROOOOOOY JEEEEENKIIIINSS!!!!

    No more job, but at least he's got chicken.

  • ratis (unregistered) in reply to Doozerboy
    Doozerboy:
    Leroy Jenkins:
    John: is everyone clear on the stored procedure replacement plan? John: Dave? ... where's Dave? ... Dave: Alright I'm back! Let's do it! John: no no wait! Dave! The plan! Dave: LEEEEEROOOOOOY JEEEEENKIIIINSS!!!!

    No more job, but at least he's got chicken.

    +1

  • The Nerve (unregistered) in reply to Anon
    Anon:
    Remy Porter:
    No, I think your reading comprehension is TRWTF.

    "With that done, he explained the next steps as he executed them: log onto production, navigate down the tree to where the stored procedures were, delete that entire branch of objects without mercy, and run the script to recreate them."

    No, your comprehension of "reply" vs "quote" is TRWTF.

    I was thinking the same thing.

  • (cs) in reply to Sam
    Sam:
    And that's why you don't teach people lessons!

    LOL! I think the original quote was "And that's why you don't teach lessons to your son."

    So... you taught me a lesson to not teach people lessons?

    I love that series...

  • sino (unregistered) in reply to The Nerve

    I disagree!

    tap tap tap tap

    SLAM!

    VRROOOOOOOOOOM!!!!

    [image]
  • (cs) in reply to rob
    rob:
    ...as he pulled the application up and logged into the test database

    TRWTF is that there was gigabytes of customer's live data in the test database.

    I think the opposite is true. Why not test on real data? Plus, they might have been able to restore from test...

  • The Nerve (unregistered) in reply to hoodaticus
    hoodaticus:
    rob:
    ...as he pulled the application up and logged into the test database

    TRWTF is that there was gigabytes of customer's live data in the test database.

    I think the opposite is true. Why not test on real data? Plus, they might have been able to restore from test...

    I hope that your system does not process credit card data or medical information. If it does, I hope I am not a customer.

  • Æâ€â„ (unregistered)

    Hey Alex, can I haz data?

    incassum: incassum not here, please leave a message. BEEP

  • The Nerve (unregistered) in reply to sino

    ...so easy to read.

  • The Nerve (unregistered) in reply to The Nerve

    Brilliant idea! A link automatically makes it...

  • (cs) in reply to Guest
    Guest:
    And nobody thought, in the history of that company, of writing a simple script which, once working, would allways delete the right tree (maybe even do some sanity-checks before doing so) and automatically start the rebuild too ?

    Nowadays, there seems to be a general aversion to expenditures of time for internal automation. The theory seems to be that it is better to spend 3 hours each time to do some manual process than to spend 30 hours to build a tool; even though the result will be thousands of hours expended on the manual process over a year and the 30 hours is expended once.

    "The cobbler's children have no shoes." We're so busy automating the world we have no time to automate our own work.

    One of my pet peeves.

  • Zombie Dave (unregistered) in reply to HAL
    HAL:
    "...just what do you think you're doing, Dave?"

    Eating your brain, HAL.

  • (cs) in reply to Jeff
    Jeff:
    "TRWTF is doing stuff like that interactively and auto-committing the transaction (or using a system without transactions in the first place). "

    I'm assuming the database is Oracle, given the use of Enterprise manager.

    DDL in Oracle auto-commits. No rollback.

    True enough, but oracle also supports:

    create or replace procedure xxx ...

  • yeah, really (unregistered) in reply to sino
    sino:
    I disagree!

    tap tap tap tap

    SLAM!

    VRROOOOOOOOOOM!!!!

    [image]

    And it was all of two posts above. Sheesh.

  • (cs) in reply to Coyne
    Coyne:
    Guest:
    And nobody thought, in the history of that company, of writing a simple script which, once working, would allways delete the right tree (maybe even do some sanity-checks before doing so) and automatically start the rebuild too ?

    Nowadays, there seems to be a general aversion to expenditures of time for internal automation. The theory seems to be that it is better to spend 3 hours each time to do some manual process than to spend 30 hours to build a tool; even though the result will be thousands of hours expended on the manual process over a year and the 30 hours is expended once.

    "The cobbler's children have no shoes." We're so busy automating the world we have no time to automate our own work.

    One of my pet peeves.

    Amusingly, I've been told off for automating things. They get forgotten about, and then when someone kicks the server, some thing that no one ever knew had to be done stops working and 20 people have to drop everything and run through the process documentation.

    The best part is when they ask the people in the department and all they know is "It used to be insert fired employees name here's job"

  • Ben (unregistered)

    Even just reading that made my heart race and sweat form on my brow.

  • No one (unregistered)

    I have to admit though that most places I've worked were seconds from disaster on any given day. The procedures for making changes in the db were ad hoc an certainly a misplaced delete could spell disaster. As far as backups I've seen long periods of neglect while the organization put all it's effort into just growing the features and keeping the lights on. Even if the backups had been decent this example would still have been a large outage and they would have likely lost several customers anyway. Sometimes it just comes down to luck.

  • uuang (unregistered)

    I like Remy's passion in the comments.

    This is the third time I've gotten amet as my captcha wtf

  • uuang (unregistered) in reply to hoodaticus
    hoodaticus:
    rob:
    ...as he pulled the application up and logged into the test database

    TRWTF is that there was gigabytes of customer's live data in the test database.

    I think the opposite is true. Why not test on real data? Plus, they might have been able to restore from test...

    Wait, whats the opposite of TRWTF being that the test database had several gigs of live data in it?

  • Æà (unregistered) in reply to uuang
    uuang:
    I like Remy's passion in the comments.

    This is the third time I've gotten amet as my captcha wtf

    The captcha system is broken. It uses a very small set of captchas. I'm surprised we're not getting more spam (or Alex is a comment moderation ninja and deletes spam before we can see it).

  • The Nerve (unregistered) in reply to uuang
    uuang:
    I like Remy's passion in the comments.

    This is the third time I've gotten amet as my captcha wtf

    I prefer Alex's (secure) attitude. He lets people blast away in the comments, and he only responds to the most blatant of issues.

  • A wise man once said (unregistered) in reply to The Nerve
    The Nerve:
    uuang:
    I like Remy's passion in the comments.

    This is the third time I've gotten amet as my captcha wtf

    I prefer Alex's (secure) attitude. He lets people blast away in the comments, and he only responds to the most blatant of issues.

    That's right Never argue with an idiot. They'll drag you down to their level and then bludgeon you to death with experience.

  • Bert Glanstron (unregistered)

    Dear Dave,

    In case you can’t tell, this is a grown-up place. The fact that you insist on deleting production data clearly shows that you’re too young and too stupid to be using the Enterprise Manager.

    Go away and grow up.

    Sincerely, Bert Glanstron

  • (cs) in reply to one space to many
    one space to many:
    I remeber a friend of mine doing something similar: 
    someone@some-computer:~/aVeryImportenetDir# rm -r oldFilesToDelete/ *

    But where did that space come in, that shouldn't be there!

    it was hilarious

    But not quite as hilarious as if the extra space had been one char to the left...

    someone@some-computer:~/aVeryImportenetDir# rm -r oldFilesToDelete /*

    (assuming sufficient privs, of course)

  • dtfhg (unregistered) in reply to John
    John:
    It always take an event like this for people to realize they *need* to test their backups every now and then... You're bound to have something like this happen if you don't.
    Yup... just last thursday our server got hacked and they basically rm -rf * .. First timeout of 20 years anything like this happened to us... Well that sucks, lets go get our backups... Oh wait, our backup server failed 3 weeks ago, and never got around to replacing it, so we temporarily mounted the drive same server that got hacked. so backups wiped... well shit.. Ok we will go to our archival optical that is a month old and use the incremental. Restore about 60% of it, but last 40% is corrupt. oh oh.. Luckily we had our databases backed up on a separate server so we only lost a days worth of that.. but for last 40% we had to go to 4 month old backups... Apparently one of the shares wasn't being backed up AT ALL, so we lost hundreds load of scripts.. hundreds of hours of programming time. We just sent drive to data recovery yesterday.. hoping we get atleast the backup files back..

    W had a bunch of emergency meetings since then and our boss said "this is everyones get out of jail free card.. if this happens again people WILL lose jobs".. so unlike Dave we didn't get any replacements.. Thank God. We sure changed our backup strategy now.

  • Grammar Nazi (unregistered) in reply to Remy Porter
    Remy Porter:
    I recommend paying attention to punctuation and grammar in the future. Those symbols actually convey some semantic information about the contents of the sentence and their relationship to the others.

    Funny you should mention it. I was...

    Remy Porter:
    Stunned by the immensity of his screw up, Dave stared at the screen, like it was the oncoming headlamps of a semi.

    Note: the comma in red is invalid.

  • MadX (unregistered) in reply to The Nerve

    I wonder if there are any WTFs about too many backups.

    facilisis: (I like that word)

  • dtfhg (unregistered) in reply to MadX
    MadX:
    I wonder if there are any WTFs about too many backups.

    facilisis: (I like that word)

    Yup. my one manager has 6 backups of his mp3s. one on his personal computer. one on his work computer, one on the local work server, one on the remote work server, one on cds, one on dvds that he keeps in his car.

  • Anon (unregistered) in reply to sino

    Yeah, 'cos it's so much easier to have to keep clicking the "reply to #random numeric code#" that actually have people quote the bit they are replying too. It's especially useful when you are browsing the comments and you spot another comment replying to 317518. Very user friendly. And I always like to get my answers before I know the question.

  • uuang (unregistered) in reply to A wise man once said
    A wise man once said:
    The Nerve:
    uuang:
    I like Remy's passion in the comments.

    This is the third time I've gotten amet as my captcha wtf

    I prefer Alex's (secure) attitude. He lets people blast away in the comments, and he only responds to the most blatant of issues.

    That's right Never argue with an idiot. They'll drag you down to their level and then bludgeon you to death with experience.

    Good point, however I still found the drama entertaining today.

    Back to work....

  • sheldon (unregistered) in reply to Jaime
    Jaime:
    BTW, the previous manager's justification for no scripts was that the developer could change a script after testing, but the deployment plan with individual steps could be printed out and therefore made unchangable.
    You should have printed the script out, put it on a wooden table...
  • Grammar Nazi (unregistered) in reply to Remy Porter
    Remy Porter:
    I recommend paying attention to punctuation and grammar in the future.

    Yeah, me too.

    Remy Porter:
    The next day, John helped Dave's replacement, Phil, reverse-engineer as much data as he could off of other servers and databases. It took weeks to complete. The company lost hundreds of thousands of dollars from the downtime alone, breached several contracts, lost several major customers and a significant chunk of their reputation.

    FTFY (Added a hyphen and removed a comma. Also fixed noun-number agreement)

    Additionally, not sure that you can "reverse-engineer" most data, but it does depend on what the data is.

  • bullshitter (unregistered)

    IRL this shit exists all over, and the first head to fly should be that of the "alfa geek" and his boss, not some lowly dipshit operator who barely know wtf he's doing. He should be just beaten with a stick and sodomized.

  • Lurker Indeed (unregistered) in reply to Uncle Al
    Uncle Al:
    For old NFL fans like myself, it was also reminiscent of Jim Mora responding to a reporter's question about his team making the playoffs.

    BACKUPS?! BACKUPS?!! DON'T TALK TO ME ABOUT BACKUPS!

    I just hope the database will start up!

    /coulda, shoulda, woulda.

  • Darth Database (unregistered) in reply to Lurker Indeed
    Lurker Indeed:
    Uncle Al:
    For old NFL fans like myself, it was also reminiscent of Jim Mora responding to a reporter's question about his team making the playoffs.

    BACKUPS?! BACKUPS?!! DON'T TALK TO ME ABOUT BACKUPS!

    I just hope the database will start up!

    /coulda, shoulda, woulda.

    I have backed up the database. Pray I don't back it up any further.

  • Darth Dave (unregistered)

    I have deleted the data...pray I don't delete the backups.

Leave a comment on “Production Promote”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article