Comment On Bourne Into Oblivion

And to think, it had nothing to do with Jason Bourne

What's an explitive?

pjt33:

What's an explitive?

Yes, let's all be pricks about spelling.

Wow, now THAT would be a great parting gift: remove all data from the entire company. How many of you out there wouldn't mind doing that as YOUR parting gift?

Congratulations on not getting the DEFCON system backwards!

Every time I hear someone describe an emergency as "DEFCON 5" I have this urge to slap them with a nuclear warhead.

I have done that.

I was trying to build a web-based content management application (this was in 1996, when web programming meant FORMs and CGI and perl) that you could use to create new web sites...and delete them too.

So I did nearly the same thing...when you wanted to delete a website, my perl script (which ran as root on the webserver) executed an "rm -rf /$path".

Needless to say, with one of the refreshes of the CGI page in my browser the $path variable was not set, and I ruined the box.

Lots of silly preventable mistakes in this story, and a lot of lessons learned. I still have a hardcopy of the offending perl CGI script, with an "F-" in red written by one of the senior developers.

Frank:

pjt33:

What's an explitive?

Yes, let's all be pricks about spelling.

It should also be "getting an error message".

So did the sh script have a large comment at the top explicitly listing all the variables that absolutely positively had to be set?

Did it have a section right at the beginning checking that they were set?

TRWTF is that this place doesn't have code reviews.

That's one hell of an oops.

That picture... isn't that the mental hospital from the movie Sling Blade?

amischiefr:

Wow, now THAT would be a great parting gift: remove all data from the entire company. How many of you out there wouldn't mind doing that as YOUR parting gift?

Actually, I would mind. And, I would like to personally beat the crap out of anyone who has ever intentionally done this when leaving a job.

Frank:

pjt33:

What's an explitive?

Yes, let's all be pricks about spelling.

That was totally not the response I was expecting to this. Of course, what I expected could still happen. Let's see how many individual words or lists of words are posted by helpful people in reply to this by the time we fill up the first page of comments.

Frist Psot:

And to think, it had nothing to do with Jason Bourne

...or DID it? Hmmmmmmm....

What would REALLY piss me off is that even if you're a responsible person fixing someone else's major screw up (And this is in fact what happened), you would forever be remembered as the person who took down the system or couldn't restore.

Nevermind that someone else didn't run backups correctly, verify that backups were stable and working condition, and that a script was running without any safeguards to ensure that the variables were in place (Call me paranoid, I rarely trust variables where file IO is concerned.. and that stems from a co-worker bringing a server down doing a recursive grep piped to a file - 800GB output, which was grepped to itself in a loop).

This isn't the first company this has happened to; it must have happened at some of Sun's customers, because they built in anti-rm-rf/ protection in the OS.

One of the cool features of Solaris 10 is that "rm -rf /" refuses to work. The Sun guys said they knew people don't type this deliberately, but often scripts intend to do "rm -rf $var1/$var2" and forget to set var1 and var2.

"Jerry was getting "thanked" for the company's current nuclear crisis and that he should have set his little "time bomb" to go off AFTER he was gone. Naturally, while his being "blamed" helped to improve the morale of everyone else, it didn't do much to help Jerry's outlook - especially since it appeared as if this was someone else's "parting gift"."


Did Jerry have to use "up" all of his "quotation" marks before he "left"?

I love how the outgoing/recently departed employee is the easy target.

Not wanting my good name dragged through the mud, I gave a trusted colleague all of my saved unfinished project mail.

Six months later over beers with former colleagues, I found out that I was being scapegoated for another division's lack of planning and pending disaster.

My saved e-mail messages outlining the risks of not planning for that particular issue (written at least a year before I left) exonorated me and left egg on the face of the accuser/actual culprit.

Someone You Know:

Congratulations on not getting the DEFCON system backwards! (It almost makes up for the middle school writing level.)

Every time I hear someone describe an emergency as "DEFCON 5" I have this urge to slap them with a nuclear warhead.

OTOH, some of us former Cold Warriors still get reflex chills when someone invokes DEFCON 1 in humor. The real thing wasn't quite so funny. Especially if you're in a CP which is a guaranteed first strike target.

The Soviet Union's been gone for 18 years. I'm almost used to the idea now.

"unpatched flaw in sudo's setup"

You mean "sudo su -"? That's not a flaw; it's a side-effect of letting a fucking idiot set up sudo.

And this is why you never let untrusted entities run as root.

That includes testing harnesses and co-ops / interns.

Joey Stink Eye Smiles:

"Jerry was getting "thanked" for the company's current nuclear crisis and that he should have set his little "time bomb" to go off AFTER he was gone. Naturally, while his being "blamed" helped to improve the morale of everyone else, it didn't do much to help Jerry's outlook - especially since it appeared as if this was someone else's "parting gift"."


Did Jerry have to use "up" all of his "quotation" marks before he "left"?

You can't say "quotation" without "quota".

Bob:

"unpatched flaw in sudo's setup"

You mean "sudo su -"? That's not a flaw; it's a side-effect of letting a fucking idiot set up sudo.

Yup. I caught that too, and got a little annoyed.

It's not an "unpatched flaw", even if it's explicitly (in the fine text) a "flaw in sudo's setup"... it's an undesired configuration option. Whoever set up the standard deployed "/etc/sudoers" obviously never really read the man page, or thought out the implications of remote root escalation... like enabling the inadvertent deletion of entire remote filesystems.

Still, it's more "dramatic" "this way". (Yup, I have a batch of """"s to use up.)

Someone You Know:

Congratulations on not getting the DEFCON system backwards! (It almost makes up for the middle school writing level.)

Every time I hear someone describe an emergency as "DEFCON 5" I have this urge to slap them with a nuclear warhead.

What's the matter? Haven't you realized that more than 99% of all the emergencies you meet is in reality jost business as usual. which makes DEFCON 5 the correct level of alertness.

Joey Stink Eye Smiles:

"Jerry was getting "thanked" for the company's current nuclear crisis and that he should have set his little "time bomb" to go off AFTER he was gone. Naturally, while his being "blamed" helped to improve the morale of everyone else, it didn't do much to help Jerry's outlook - especially since it appeared as if this was someone else's "parting gift"."


Did Jerry have to use "up" all of his "quotation" marks before he "left"?

No, but he did have to nest them correctly. (Hint: use " for external quotation marks, ' for internal quotation marks, or vice-versa)

Someone You Know:

Congratulations on not getting the DEFCON system backwards!

Every time I hear someone describe an emergency as "DEFCON 5" I have this urge to slap them with a nuclear warhead.

Well, if someone tells you that the current emergency is "DEFCON 5", simply treat is as such and go back to sleep.

Yeah... sad to say... this happened at a large telecom software company I worked for. Sadly, it was worse because we had automount set up and the script in question as it traversed the directory structure was mounting just about anything in the automount map. Anything that had world rwx was getting nuked.

Sent 3500+ people home for a day and subsequent weekend to get stuff restored from tape. Luckily, we had the tapes.

Stuart:

This isn't the first company this has happened to; it must have happened at some of Sun's customers, because they built in anti-rm-rf/ protection in the OS.

One of the cool features of Solaris 10 is that "rm -rf /" refuses to work. The Sun guys said they knew people don't type this deliberately, but often scripts intend to do "rm -rf $var1/$var2" and forget to set var1 and var2.

I wonder how many people reading this want to try it out to see what error message is displayed?

Don't push the big red button.

What does SNAFU'd stand for?

Bosluis:

What does SNAFU'd stand for?

Here you go.

Old Soldier:

The Soviet Union's been gone for 18 years. I'm almost used to the idea now.

Don't get too used to. Putin is building Soviet Union 2.0.

Ken B:

Someone You Know:

Congratulations on not getting the DEFCON system backwards!

Every time I hear someone describe an emergency as "DEFCON 5" I have this urge to slap them with a nuclear warhead.

Well, if someone tells you that the current emergency is "DEFCON 5", simply treat is as such and go back to sleep.

We're at DEFCON-6 here people! Go grab a beer and take a nap.

The original script developer is a moron, not the co-op, some sanity checks on the variables before a recursive, forced, rm would be ABSOLUTELY NECESSARY.

if [[ -z $var1 || -z $var2 ]]; then

  echo -e "\aOnly a fool would forget to specify \$var1 or \$var2, I shall abuse you verbally."

else

  rm -rf $var1/$var2

fi

The first rule of shell scripting is set -u

The second rule of shell scripting is rewrite it in perl

This reminds me of an incident in my first ever job as a helpdesk tech. I got a panicked call from a lady in accounts saying that all her data had "just gone". "Vanished". "Help!".

So I went to her desk and asked her to explain firstly what the data was from, where it was stored and then what she had been doing before it "vanished". She explained that some absolutely vital section of the company accounts were maintained on a legacy system on a standalone 286 PC. Because it wasn't networked, and because of the business-critical nature of the data it contained, my predecessor had taught her how to regularly back-up the data to a floppy disk, but she confessed that because it was year-end she hadn't had time to do the backup for a few weeks, so weeks worth of hard work had gone. And ironically, the data actually vanished whilst she was performing the backup!

So I asked her to demonstrate the steps she'd been doing at the DOS prompt, but without pressing the enter key unless I said it was OK so that I could be sure of maximising my chance of an undelete. She showed me how she changed folder into the place where all the accounts data was stored, did a directory listing, and then copied all the files to her backup 3.5" floppy. "Oh, but I did some housekeeping first," she remembered. And then pointed at a file in the directory listing that she had been trying to get rid of when all the data went missing. "It's the one named dot, it's still there look!". She'd typed "del ." followed by an unthinking "Y".

A little knowledge can be a dangerous thing!

Y'know, being root on one machine doesn't give you any special access to an drive NFS mounted from another machine. And I can't imagine why the script would become root to do the cleanup. I enjoy a good story as much as the next guy, and I like the $var1/$var2 "hook", but I'm thinking this story was invented to go with the hook after somebody thought it up.

My favorite mistake (which I see far too frequently) is something like:

cd /some/directory/which/we/presume/is/there

rm -rf *

Yes, works ok when the directory is there, but what if it isn't?

Always include some conditional on a cd:

cd /some/dir || exit 1

or some other abort even if the unexpected happens.

Somebody familiar with the Bourne shell, please show us teh codez to check for this. :-X I think the following *might* (unless my Google results are colliding with Bash) check for empty strings,...

if [ -z "$var1" ]; then

    echo "var1 is not set! OMFG, what are you DOING?!"

    exit -1

elif [ -z "$var2" ]; then

    echo "var2 is not set! OMFG, what are you DOING?!"

    exit -1

fi

...but what if var1 and var2 happen to be nothing more than whitespace (i.e., ' ')?

LOUD NOISES

Stig:

LOUD NOISES

BANG!

Pedantic:

The original script developer is a moron, not the co-op, some sanity checks on the variables before a recursive, forced, rm would be ABSOLUTELY NECESSARY.

if [[ -z $var1 || -z $var2 ]]; then

  echo -e "\aOnly a fool would forget to specify \$var1 or \$var2, I shall abuse you verbally."

else

  rm -rf $var1/$var2

fi

I'm very glad you saw fit to explain that; I was just about to waste several precious seconds by either (a) trying to remember how to check a variable in a shell script or (b) typing "bourne shell tutorial" into Google.

Otherwise: if you make something idiot-proof, somebody will come up with a better idiot. This is a superb WTF, with minimal verbosity. I particularly love the idea of "sudo su -", which has got to be the shortest possible sysadmin fuckup of all time. (Alternative candidates solicited...)

EFH:

Y'know, being root on one machine doesn't give you any special access to an drive NFS mounted from another machine.

I suspect it does if a brillant Expert somewhere sets it up to.

EFH:

And I can't imagine why the script would become root to do the cleanup.

Obviously, the original developer didn't want to be bothered with making sure permissions are set correctly.

EFH:

...I'm thinking this story was invented to go with the hook after somebody thought it up.

Entirely possible, but not the only explanation.

xtremezone:

Somebody familiar with the Bourne shell, please show us teh codez to check for this. :-X I think the following *might* (unless my Google results are colliding with Bash) check for empty strings,...

if [ -z "$var1" ]; then

    echo "var1 is not set! OMFG, what are you DOING?!"

    exit -1

elif [ -z "$var2" ]; then

    echo "var2 is not set! OMFG, what are you DOING?!"

    exit -1

fi

...but what if var1 and var2 happen to be nothing more than whitespace (i.e., ' ')?

Classically (and it works across all vintages of all shell scripts), append an 'x' to the target and compare against "x".

Non-classically, do it properly. In Perl.

if [ -z "`echo \"$var1\" | sed -r 's/^\\s+//g;s/\s+\$//g'`" ]; then

    echo "How should I know. Look at that thing."

    exit -1

fi

???

Bim Job:

xtremezone:

Somebody familiar with the Bourne shell, please show us teh codez to check for this. :-X I think the following *might* (unless my Google results are colliding with Bash) check for empty strings,...

if [ -z "$var1" ]; then

    echo "var1 is not set! OMFG, what are you DOING?!"

    exit -1

elif [ -z "$var2" ]; then

    echo "var2 is not set! OMFG, what are you DOING?!"

    exit -1

fi

...but what if var1 and var2 happen to be nothing more than whitespace (i.e., ' ')?

Classically (and it works across all vintages of all shell scripts), append an 'x' to the target and compare against "x".

Non-classically, do it properly. In Perl.

Ah yes, I forgot about that trick. :-[ Thank you.

if [[ -z $var1 || -z $var2 ]]; then

You want double-quotes around those variables, otherwise you'll get a syntax error because $var1 just disappears into nothingness when its value is empty.

And I don't think you want to trust your sh to *not* somehow execute the else condition when that happens.

Pedantic:

The original script developer is a moron, not the co-op, some sanity checks on the variables before a recursive, forced, rm would be ABSOLUTELY NECESSARY.

if [[ -z $var1 || -z $var2 ]]; then

  echo -e "\aOnly a fool would forget to specify \$var1 or \$var2, I shall abuse you verbally."

else

  rm -rf $var1/$var2

fi

What? Are you going to activate the speech synthesiser and tell the user their mother was a hamster and their father smelt of elderberries?

Skizz

Stuart:

This isn't the first company this has happened to; it must have happened at some of Sun's customers, because they built in anti-rm-rf/ protection in the OS.

One of the cool features of Solaris 10 is that "rm -rf /" refuses to work. The Sun guys said they knew people don't type this deliberately, but often scripts intend to do "rm -rf $var1/$var2" and forget to set var1 and var2.

That protection has been put into almost all unixes, including Linux. Log in as root and try it if you don't believe me.

Dan:

I love how the outgoing/recently departed employee is the easy target.

Not wanting my good name dragged through the mud, I gave a trusted colleague all of my saved unfinished project mail.

Six months later over beers with former colleagues, I found out that I was being scapegoated for another division's lack of planning and pending disaster.

My saved e-mail messages outlining the risks of not planning for that particular issue (written at least a year before I left) exonorated me and left egg on the face of the accuser/actual culprit.

Passing blame to employees who have left the company is a sad tactic by bad management to get out of taking responsibility for their own screw ups. Same thing happened to me, even though I passed as much of my knowledge onto my fellow developers as possible before leaving. They ended up cancelling my project soon after, blaming me of course. Yet, instead of taking my 2 weeks notice and getting all the info they required out of me, they shipped me out the door the next day. When this sorta thing happens, you have to know that the SMART people where you worked won't turn on you, because they know who is REALLY at fault.

Ken B:

Someone You Know:

Congratulations on not getting the DEFCON system backwards!

Every time I hear someone describe an emergency as "DEFCON 5" I have this urge to slap them with a nuclear warhead.

Well, if someone tells you that the current emergency is "DEFCON 5", simply treat is as such and go back to sleep.

You know, in Britain it's called the BIKINI system. DEFCON sounds so much more intimidating than "hallo guvna, we're at bikini red, yes we are." "Righto, chap."

Frank:

What would REALLY piss me off is that even if you're a responsible person fixing someone else's major screw up (And this is in fact what happened), you would forever be remembered as the person who took down the system or couldn't restore.

People always blame the messenger. It's in our nature.