- Feature Articles
- CodeSOD
- Error'd
-
Forums
-
Other Articles
- Random Article
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
I accidentally deleted the rest of
No worries. I can restore the comment from backup!
TRWTF is a brand new hire "training" the replacement.
Admin
When deleting something takes even a tiny bit longer than you expect, that means you are deleting more than you expected. CTRL-C!
Something like this happened to me just last night, but fortunately all I deleted was the local copy of a CVS repository, losing some work but not tons and tons of data.
Admin
It always take an event like this for people to realize they need to test their backups every now and then... You're bound to have something like this happen if you don't.
Admin
I would wager, sadly, that this is how the majority of "mission critical" systems operate. As always, the root cause is not the developer but incompetent management who have no idea of the right ways to do things.
Admin
So we've moved on from unicorns to LOLTrek. I'm not sure if that's an improvement or not.
Admin
No, TRWTF is the lame Star Trek joke in the gif.
Admin
And nobody thought, in the history of that company, of writing a simple script which, once working, would allways delete the right tree (maybe even do some sanity-checks before doing so) and automatically start the rebuild too ?
Admin
Rookie mistake. Never shake the scheduler on the production server.
Admin
How hard would it be to drop the stupid procedures as part of the script? Srsly. If any of your daily tasks takes more than one step, you are inviting whoopsies.
Admin
One word: Ouch!
It's stories like these that make you sit up in disbelief at the practices of some individuals and make you want to go over your disaster recovery plan.
Captcha: Plaga - as in Dave was a plaga to his former company.
Admin
The thing I've noticed is that most of the time, there's no time or budget to HAVE a disaster recovery plan, so there's either a half-assed one put in place, or nothing at all.
Again, management is the problem.
Admin
Where I work, our deployment processes are begging for this to happen. The previous manager demanded that all deployment steps be executed by the user, scripting or automating the steps was considered obfuscation and not allowed. So, we regularly had deployments that involved copying hundreds of artifacts to productions systems and manipulating them by hand. Only very recently have I been able to convince management that an escrowed script is way more reliable and testable than a piece of paper.
BTW, the previous manager's justification for no scripts was that the developer could change a script after testing, but the deployment plan with individual steps could be printed out and therefore made unchangable. Apparently the manager had not heard of file-system security.
Admin
Haha, from the source code:
<!-- Dave whipped around in the chair so quickly that he sliced his arm off on the edge of a server rack. He screamed as blood pumped from the stump; John screamed. Suddenly, Dave stopped screaming, the fake blood stopped pumping, he grabbed John by the collar, and said, "And *that's why you have a robust install process for production!*" -->Also
<!-- And that's why you don't teach people lessons! Sorry for the pop-culture extravaganza. Okay, not really. --><!-- Easy Reader Version: See Dave. See Dave perform without a net. See Dave break his neck. See Phil get a promotion. -->TRWTF is who is Phil?!
Admin
Hmm. An Oncoming Semi? Isn't that a contradiction in terms?
Admin
Nononono. Wait for it to finish then CTRL-Z. Cooler heads prevail.
Admin
Did you even read the story? Phil was Dave's replacement. He also wasn't a new hire, evidently, but another developer on the team who got promoted to the role Dave was in (that part was to "The Nerve" who asked why the new guy would train a replacement).
Admin
Admin
So... I guess John wasn't watching too closely either.
Admin
"...just what do you think you're doing, Dave?"
Admin
As bad as this sounds, nothing beats the thrill of performing without a net.
Admin
TRWTF is doing stuff like that interactively and auto-committing the transaction (or using a system without transactions in the first place).
CAPTCHA: "persto" - Presto! Rolled that back for you.
Admin
I'm sorry, Dave. I can't let you do that.
Admin
It's a new author, I think. I like him.
Admin
Admin
I know, right? I mean, there were two of them ... they should have been practicing XP!!
Oh, wait ... they were ... eXperimenting with Production.
Admin
Standard SQL- DDL statements never happen in a transaction. "DROP TABLE" can't be rolled back.
Admin
^^^ This, I'd extend the script to first make a backup before deployment as well.
Admin
Still, I think management is to blame. They usually are.
Admin
I kept waiting for a John Dies At The End joke.
Admin
Whoa! I absentmindedly selected some text and the next thing you know Kirk and Data are doing the old front and back with a unicorn.
Admin
Be afraid. Be very afraid.
Admin
kinda ironic that I saw an advertisement today for Red Gate's SQL Source Control tool that can be integrated into Enterprise manager. I'm sure stuff like this happens all the time.
Admin
click on gigabytes
Admin
We have all made screw ups, it seems there was not proper Diaster Recovery in place, and I actually feel sorry for Dave.
Of course standard practive is usually make a backup before doing a push to live.
Admin
It really depends on your system; Postgres has Transactional DDL for example. But that just reinforces my point: If your system doesn't allow you to rollback DDL, you simply shouldn't issue (inherently untested) DDL commands interactively. Well, not on production anyway.
Admin
So, with some quotes in hand and a few idea's on how we would keep data synchronized; I walked into the breach.
'Sorry Donny, we'll have to pass on the DR Cold Site. Was reading an article last night about how it's a real white elephant, and we don't have the money right now for things that are not critical path'
[Which critical path usually meant the CEO's new private jet]
A few months later .. yeah it happened. Our systems died horribly. Seems I was supposed to advise on ways around this, and hey, why didn't we have a Cold Storage that could have been fired up in this case..
I think he still works there; and still reads 'Executive IT Management' magazines.
Admin
The first lines of any code for a replacement stored procedure should be to see if an old one exists, and drop it if it does.
CAPTCHA: odio(-delayheehooo)
Admin
"TRWTF is doing stuff like that interactively and auto-committing the transaction (or using a system without transactions in the first place). "
I'm assuming the database is Oracle, given the use of Enterprise manager.
DDL in Oracle auto-commits. No rollback.
Admin
Why not take a backup immediately before deleting anything on production? Just a thought.
Admin
Me: I can haz first post? Alex: No! LOL deletes post
Admin
Gah!
Thank god for using postgres!:
=>CREATE TABLE test (id serial, name VARCHAR(32)); CREATE TABLE =>INSERT INTO test (name) VALUES ('testval'); INSERT 0 1 =>SELECT * FROM test; id | name
----+--------- 1 | testval (1 row)
=>BEGIN; BEGIN =>DROP TABLE test; DROP TABLE => SELECT * FROM test; ERROR: relation "test" does not exist => ROLLBACK; ROLLBACK => SELECT * FROM test; id | name
----+--------- 1 | testval (1 row)
Admin
hahaha... The lack of backups is a huge WTF... And it's not like that doesn't happen all over the place I suspect...
Worked at a place once where we had 'nightly backups, that get rolled into weekly backups, then to monthly'...
Until the first time we actually needed one, only to find that, yes, each night a backup file was created, and at the end of a week, the weekly was created, etc, etc, except that they were all empty. A beautiful folder tree of empty files.
Seems that once all the scheduling logic had been worked out and was behaving, the responsible party had neglected to uncomment the part of the script where all the real backup work was done.
Admin
Admin
Yep. What he SHOULD have done was put a DDL trigger on the production server to prevent people from dropping important stuf... oh wait... this was SQL 2000 right? Scratch that idea then.
Admin
Admin
Nice easter egg in the picture... (wait for a minute or so)
Admin
Oracle has had the "FLASHBACK TABLE foo TO BEFORE DROP" command since 10g.
Still doesn't help if you did a TRUNCATE TABLE of course.
Admin
Thats a classic, its like leaving the 'where' statement out of an update querey.
Admin
Ah, I feel bad for this guy. Most of these stories involve arrogance or unwillingness to learn / stupidity. And, while the latter is there for sure - we've all made mistakes - I've accidentally deleted a file or two.
But, it is pretty bad not to be backing up data worth hundreds of thousands of dollars.
Admin
You mean the one everyone else has noticed ad nauseum?