Comment On Production Promote

I accidentally deleted the rest of

No worries. I can restore the comment from backup!

TRWTF is a brand new hire "training" the replacement.

When deleting something takes even a tiny bit longer than you expect, that means you are deleting more than you expected. CTRL-C!

Something like this happened to me just last night, but fortunately all I deleted was the local copy of a CVS repository, losing some work but not tons and tons of data.

It always take an event like this for people to realize they *need* to test their backups every now and then... You're bound to have something like this happen if you don't.

I would wager, sadly, that this is how the majority of "mission critical" systems operate. As always, the root cause is not the developer but incompetent management who have no idea of the right ways to do things.

So we've moved on from unicorns to LOLTrek. I'm not sure if that's an improvement or not.

No, TRWTF is the lame Star Trek joke in the gif.

And nobody thought, in the history of that company, of writing a simple script which, once working, would allways delete the right tree (maybe even do some sanity-checks before doing so) and automatically start the rebuild too ?

Remy Porter:

The backup maintenance task, also Dave's responsibility, had fallen out of the scheduler...

Rookie mistake. Never shake the scheduler on the production server.

How hard would it be to drop the stupid procedures as part of the script? Srsly. If any of your daily tasks takes more than one step, you are inviting whoopsies.

One word: Ouch!

It's stories like these that make you sit up in disbelief at the practices of some individuals and make you want to go over your disaster recovery plan.

Captcha: Plaga - as in Dave was a plaga to his former company.

The thing I've noticed is that most of the time, there's no time or budget to HAVE a disaster recovery plan, so there's either a half-assed one put in place, or nothing at all.

Again, management is the problem.

Where I work, our deployment processes are begging for this to happen. The previous manager demanded that all deployment steps be executed by the user, scripting or automating the steps was considered obfuscation and not allowed. So, we regularly had deployments that involved copying hundreds of artifacts to productions systems and manipulating them by hand. Only very recently have I been able to convince management that an escrowed script is way more reliable and testable than a piece of paper.

BTW, the previous manager's justification for no scripts was that the developer could change a script after testing, but the deployment plan with individual steps could be printed out and therefore made unchangable. Apparently the manager had not heard of file-system security.

Haha, from the source code:

<!-- Dave whipped around in the chair so quickly that he sliced his arm off on the edge of a server rack. He screamed as blood pumped from the stump; John screamed. Suddenly, Dave stopped screaming, the fake blood stopped pumping, he grabbed John by the collar, and said, "And *that's why you have a robust install process for production!*" -->

Also

<!-- And that's why you don't teach people lessons! Sorry for the pop-culture extravaganza. Okay, not really. --><!-- Easy Reader Version: See Dave. See Dave perform without a net. See Dave break his neck. See Phil get a promotion. -->

TRWTF is who is Phil?!

Hmm. An Oncoming Semi? Isn't that a contradiction in terms?

Arancaytar:

When deleting something takes even a tiny bit longer than you expect, that means you are deleting more than you expected. CTRL-C!

Nononono. Wait for it to finish then CTRL-Z. Cooler heads prevail.

Sam:

TRWTF is who is Phil?!

Did you even read the story? Phil was Dave's replacement. He also wasn't a new hire, evidently, but another developer on the team who got promoted to the role Dave was in (that part was to "The Nerve" who asked why the new guy would train a replacement).

Guest:

And nobody thought, in the history of that company, of writing a simple script which, once working, would allways delete the right tree (maybe even do some sanity-checks before doing so) and automatically start the rebuild too ?

+1

So... I guess John wasn't watching too closely either.

As bad as this sounds, nothing beats the thrill of performing without a net.

TRWTF is doing stuff like that interactively and auto-committing the transaction (or using a system without transactions in the first place).

CAPTCHA: "persto" - Presto! Rolled that back for you.

HAL:

"...just what do you think you're doing, Dave?"

I'm sorry, Dave. I can't let you do that.

Sam:

Haha, from the source code:

<!-- Dave whipped around in the chair so quickly that he sliced his arm off on the edge of a server rack. He screamed as blood pumped from the stump; John screamed. Suddenly, Dave stopped screaming, the fake blood stopped pumping, he grabbed John by the collar, and said, "And *that's why you have a robust install process for production!*" -->

Also

<!-- And that's why you don't teach people lessons! Sorry for the pop-culture extravaganza. Okay, not really. --><!-- Easy Reader Version: See Dave. See Dave perform without a net. See Dave break his neck. See Phil get a promotion. -->

TRWTF is who is Phil?!

It's a new author, I think. I like him.

HAL:

"...just what do you think you're doing, Dave?"

+100

JamesQMurphy:

So... I guess John wasn't watching too closely either.

I know, right? I mean, there were two of them ... they should have been practicing XP!!

Oh, wait ... they were ... eXperimenting with Production.

Grunt:

TRWTF is doing stuff like that interactively and auto-committing the transaction (or using a system without transactions in the first place).

Standard SQL- DDL statements never happen in a transaction. "DROP TABLE" can't be rolled back.

^^^ This, I'd extend the script to first make a backup before deployment as well.

John:

It always take an event like this for people to realize they *need* to test their backups every now and then... You're bound to have something like this happen if you don't.

Backup testing was probably one of Dave's responsibilities, too.

Still, I think management is to blame. They usually are.

I kept waiting for a John Dies At The End joke.

Whoa! I absentmindedly selected some text and the next thing you know Kirk and Data are doing the old front and back with a unicorn.

Be afraid. Be very afraid.

kinda ironic that I saw an advertisement today for Red Gate's SQL Source Control tool that can be integrated into Enterprise manager. I'm sure stuff like this happens all the time.

click on gigabytes

We have all made screw ups, it seems there was not proper Diaster Recovery in place, and I actually feel sorry for Dave.

Of course standard practive is usually make a backup before doing a push to live.

Remy Porter:

Standard SQL- DDL statements never happen in a transaction. "DROP TABLE" can't be rolled back on all databases.

FTFY

It really depends on your system; Postgres has Transactional DDL for example. But that just reinforces my point: If your system doesn't allow you to rollback DDL, you simply shouldn't issue (inherently untested) DDL commands interactively. Well, not on production anyway.

ObiWayneKenobi:

I would wager, sadly, that this is how the majority of "mission critical" systems operate. As always, the root cause is not the developer but incompetent management who have no idea of the right ways to do things.

Well.. actually.. I find it's usually the management that ARE TOLD the right way to do things; but steadfastly deny that it's important, since "it never caused a problem before". Once upon a time I had a manager that loved reading 'Executive IT magazines' - you know, those ones written by people with MBA's and no IT knowledge whatsoever.
Thick as a couple of planks when it came to IT, but it was the 90's and nobody cared so long as the manager could make Gantt charts. I had walked into his office the previous morning to discuss the budget for a disaster recovery cold site, since we were running 'mission critical' systems for our international business.
'No time today, come back tomorrow'

So, with some quotes in hand and a few idea's on how we would keep data synchronized; I walked into the breach.

'Sorry Donny, we'll have to pass on the DR Cold Site. Was reading an article last night about how it's a real white elephant, and we don't have the money right now for things that are not critical path'
[Which critical path usually meant the CEO's new private jet]

A few months later .. yeah it happened. Our systems died horribly. Seems _I_ was supposed to advise on ways around this, and hey, why didn't we have a Cold Storage that could have been fired up in this case..

I think he still works there; and still reads 'Executive IT Management' magazines.

The first lines of any code for a replacement stored procedure should be to see if an old one exists, and drop it if it does.

CAPTCHA: odio(-delayheehooo)

"TRWTF is doing stuff like that interactively and auto-committing the transaction (or using a system without transactions in the first place). "

I'm assuming the database is Oracle, given the use of Enterprise manager.

DDL in Oracle auto-commits. No rollback.

Why not take a backup immediately before deleting anything on production? Just a thought.

Me: I can haz first post?
Alex: No! LOL *deletes post*

Remy Porter:

Grunt:

TRWTF is doing stuff like that interactively and auto-committing the transaction (or using a system without transactions in the first place).

Standard SQL- DDL statements never happen in a transaction. "DROP TABLE" can't be rolled back.

Gah!

Thank god for using postgres!:

=>CREATE TABLE test (id serial, name VARCHAR(32));
CREATE TABLE
=>INSERT INTO test (name) VALUES ('testval');
INSERT 0 1
=>SELECT * FROM test;
id | name
----+---------
1 | testval
(1 row)

=>BEGIN;
BEGIN
=>DROP TABLE test;
DROP TABLE
=> SELECT * FROM test;
ERROR: relation "test" does not exist
=> ROLLBACK;
ROLLBACK
=> SELECT * FROM test;
id | name
----+---------
1 | testval
(1 row)

hahaha... The lack of backups is a huge WTF... And it's not like that doesn't happen all over the place I suspect...

Worked at a place once where we had 'nightly backups, that get rolled into weekly backups, then to monthly'...

Until the first time we actually needed one, only to find that, yes, each night a backup file was created, and at the end of a week, the weekly was created, etc, etc, except that they were all empty. A beautiful folder tree of empty files.

Seems that once all the scheduling logic had been worked out and was behaving, the responsible party had neglected to uncomment the part of the script where all the real backup work was done.

Someone You Know:

So we've moved on from unicorns to LOLTrek. I'm not sure if that's an improvement or not.

Oh, there's still unicorns. Just click the word "data" near the image.

Remy Porter:

Grunt:

TRWTF is doing stuff like that interactively and auto-committing the transaction (or using a system without transactions in the first place).

Standard SQL- DDL statements never happen in a transaction. "DROP TABLE" can't be rolled back.

Yep. What he SHOULD have done was put a DDL trigger on the production server to prevent people from dropping important stuf... oh wait... this was SQL 2000 right? Scratch that idea then.

Daisy Daisy,
Give me your answer do!
I'm half crazy,
All for the love of you!
It won't be a stylish marriage,
I can't afford a carriage,
But you'll look sweet on the seat
Of a bicycle built for two !

Nice easter egg in the picture... (wait for a minute or so)

Grunt:

Remy Porter:

Standard SQL- DDL statements never happen in a transaction. "DROP TABLE" can't be rolled back on all databases.

FTFY

It really depends on your system; Postgres has Transactional DDL for example. But that just reinforces my point: If your system doesn't allow you to rollback DDL, you simply shouldn't issue (inherently untested) DDL commands interactively. Well, not on production anyway.

Oracle has had the "FLASHBACK TABLE foo TO BEFORE DROP" command since 10g.

Still doesn't help if you did a TRUNCATE TABLE of course.

Thats a classic, its like leaving the 'where' statement out of an update querey.

Ah, I feel bad for this guy. Most of these stories involve arrogance or unwillingness to learn / stupidity. And, while the latter is there for sure - we've all made mistakes - I've accidentally deleted a file or two.

But, it is pretty bad not to be backing up data worth hundreds of thousands of dollars.

You mean the one everyone else has noticed ad nauseum?