- Feature Articles
- CodeSOD
-
Error'd
- Most Recent Articles
- Stop Poking Me!
- Operation Erred Successfully
- A Dark Turn
- Nothing Doing
- Home By Another Way
- Coast Star
- Forsooth
- Epic
- Forums
-
Other Articles
- Random Article
- Other Series
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
And to think, it had nothing to do with Jason Bourne
Admin
What's an explitive?
Admin
Yes, let's all be pricks about spelling.
Admin
Wow, now THAT would be a great parting gift: remove all data from the entire company. How many of you out there wouldn't mind doing that as YOUR parting gift?
Admin
Congratulations on not getting the DEFCON system backwards!
Every time I hear someone describe an emergency as "DEFCON 5" I have this urge to slap them with a nuclear warhead.
Admin
I have done that.
I was trying to build a web-based content management application (this was in 1996, when web programming meant FORMs and CGI and perl) that you could use to create new web sites...and delete them too.
So I did nearly the same thing...when you wanted to delete a website, my perl script (which ran as root on the webserver) executed an "rm -rf /$path".
Needless to say, with one of the refreshes of the CGI page in my browser the $path variable was not set, and I ruined the box.
Lots of silly preventable mistakes in this story, and a lot of lessons learned. I still have a hardcopy of the offending perl CGI script, with an "F-" in red written by one of the senior developers.
Admin
Admin
So did the sh script have a large comment at the top explicitly listing all the variables that absolutely positively had to be set?
Did it have a section right at the beginning checking that they were set?
TRWTF is that this place doesn't have code reviews.
Admin
That's one hell of an oops.
Admin
That picture... isn't that the mental hospital from the movie Sling Blade?
Admin
Actually, I would mind. And, I would like to personally beat the crap out of anyone who has ever intentionally done this when leaving a job.
Admin
That was totally not the response I was expecting to this. Of course, what I expected could still happen. Let's see how many individual words or lists of words are posted by helpful people in reply to this by the time we fill up the first page of comments.
Admin
Admin
What would REALLY piss me off is that even if you're a responsible person fixing someone else's major screw up (And this is in fact what happened), you would forever be remembered as the person who took down the system or couldn't restore.
Nevermind that someone else didn't run backups correctly, verify that backups were stable and working condition, and that a script was running without any safeguards to ensure that the variables were in place (Call me paranoid, I rarely trust variables where file IO is concerned.. and that stems from a co-worker bringing a server down doing a recursive grep piped to a file - 800GB output, which was grepped to itself in a loop).
Admin
This isn't the first company this has happened to; it must have happened at some of Sun's customers, because they built in anti-rm-rf/ protection in the OS.
One of the cool features of Solaris 10 is that "rm -rf /" refuses to work. The Sun guys said they knew people don't type this deliberately, but often scripts intend to do "rm -rf $var1/$var2" and forget to set var1 and var2.
Admin
"Jerry was getting "thanked" for the company's current nuclear crisis and that he should have set his little "time bomb" to go off AFTER he was gone. Naturally, while his being "blamed" helped to improve the morale of everyone else, it didn't do much to help Jerry's outlook - especially since it appeared as if this was someone else's "parting gift"."
Did Jerry have to use "up" all of his "quotation" marks before he "left"?
Admin
I love how the outgoing/recently departed employee is the easy target.
Not wanting my good name dragged through the mud, I gave a trusted colleague all of my saved unfinished project mail.
Six months later over beers with former colleagues, I found out that I was being scapegoated for another division's lack of planning and pending disaster.
My saved e-mail messages outlining the risks of not planning for that particular issue (written at least a year before I left) exonorated me and left egg on the face of the accuser/actual culprit.
Admin
OTOH, some of us former Cold Warriors still get reflex chills when someone invokes DEFCON 1 in humor. The real thing wasn't quite so funny. Especially if you're in a CP which is a guaranteed first strike target.
The Soviet Union's been gone for 18 years. I'm almost used to the idea now.
Admin
"unpatched flaw in sudo's setup"
You mean "sudo su -"? That's not a flaw; it's a side-effect of letting a fucking idiot set up sudo.
Admin
And this is why you never let untrusted entities run as root.
That includes testing harnesses and co-ops / interns.
Admin
You can't say "quotation" without "quota".
Admin
Yup. I caught that too, and got a little annoyed.
It's not an "unpatched flaw", even if it's explicitly (in the fine text) a "flaw in sudo's setup"... it's an undesired configuration option. Whoever set up the standard deployed "/etc/sudoers" obviously never really read the man page, or thought out the implications of remote root escalation... like enabling the inadvertent deletion of entire remote filesystems.
Still, it's more "dramatic" "this way". (Yup, I have a batch of """"s to use up.)
Admin
What's the matter? Haven't you realized that more than 99% of all the emergencies you meet is in reality jost business as usual. which makes DEFCON 5 the correct level of alertness.
Admin
No, but he did have to nest them correctly. (Hint: use " for external quotation marks, ' for internal quotation marks, or vice-versa)
Admin
Admin
Yeah... sad to say... this happened at a large telecom software company I worked for. Sadly, it was worse because we had automount set up and the script in question as it traversed the directory structure was mounting just about anything in the automount map. Anything that had world rwx was getting nuked.
Sent 3500+ people home for a day and subsequent weekend to get stuff restored from tape. Luckily, we had the tapes.
Admin
I wonder how many people reading this want to try it out to see what error message is displayed?
Don't push the big red button.
Admin
What does SNAFU'd stand for?
Admin
Admin
We're at DEFCON-6 here people! Go grab a beer and take a nap.
Admin
The original script developer is a moron, not the co-op, some sanity checks on the variables before a recursive, forced, rm would be ABSOLUTELY NECESSARY.
Admin
The first rule of shell scripting is set -u
The second rule of shell scripting is rewrite it in perl
Admin
This reminds me of an incident in my first ever job as a helpdesk tech. I got a panicked call from a lady in accounts saying that all her data had "just gone". "Vanished". "Help!".
So I went to her desk and asked her to explain firstly what the data was from, where it was stored and then what she had been doing before it "vanished". She explained that some absolutely vital section of the company accounts were maintained on a legacy system on a standalone 286 PC. Because it wasn't networked, and because of the business-critical nature of the data it contained, my predecessor had taught her how to regularly back-up the data to a floppy disk, but she confessed that because it was year-end she hadn't had time to do the backup for a few weeks, so weeks worth of hard work had gone. And ironically, the data actually vanished whilst she was performing the backup!
So I asked her to demonstrate the steps she'd been doing at the DOS prompt, but without pressing the enter key unless I said it was OK so that I could be sure of maximising my chance of an undelete. She showed me how she changed folder into the place where all the accounts data was stored, did a directory listing, and then copied all the files to her backup 3.5" floppy. "Oh, but I did some housekeeping first," she remembered. And then pointed at a file in the directory listing that she had been trying to get rid of when all the data went missing. "It's the one named dot, it's still there look!". She'd typed "del ." followed by an unthinking "Y".
A little knowledge can be a dangerous thing!
Admin
Y'know, being root on one machine doesn't give you any special access to an drive NFS mounted from another machine. And I can't imagine why the script would become root to do the cleanup. I enjoy a good story as much as the next guy, and I like the $var1/$var2 "hook", but I'm thinking this story was invented to go with the hook after somebody thought it up.
Admin
My favorite mistake (which I see far too frequently) is something like:
Yes, works ok when the directory is there, but what if it isn't?
Always include some conditional on a cd:
or some other abort even if the unexpected happens.
Admin
Somebody familiar with the Bourne shell, please show us teh codez to check for this. :-X I think the following might (unless my Google results are colliding with Bash) check for empty strings,...
...but what if var1 and var2 happen to be nothing more than whitespace (i.e., ' ')?
Admin
LOUD NOISES
Admin
BANG!
Admin
Otherwise: if you make something idiot-proof, somebody will come up with a better idiot. This is a superb WTF, with minimal verbosity. I particularly love the idea of "sudo su -", which has got to be the shortest possible sysadmin fuckup of all time. (Alternative candidates solicited...)
Admin
Admin
Non-classically, do it properly. In Perl.
Admin
???
Admin
Admin
And I don't think you want to trust your sh to not somehow execute the else condition when that happens.
Admin
What? Are you going to activate the speech synthesiser and tell the user their mother was a hamster and their father smelt of elderberries?
Skizz
Admin
Admin
Passing blame to employees who have left the company is a sad tactic by bad management to get out of taking responsibility for their own screw ups. Same thing happened to me, even though I passed as much of my knowledge onto my fellow developers as possible before leaving. They ended up cancelling my project soon after, blaming me of course. Yet, instead of taking my 2 weeks notice and getting all the info they required out of me, they shipped me out the door the next day. When this sorta thing happens, you have to know that the SMART people where you worked won't turn on you, because they know who is REALLY at fault.
Admin
You know, in Britain it's called the BIKINI system. DEFCON sounds so much more intimidating than "hallo guvna, we're at bikini red, yes we are." "Righto, chap."
Admin
People always blame the messenger. It's in our nature.
Admin
I had no idea it was called BIKINI. They even dropped the level from red to amber yesterday but the news just reported it as going from "severe" to "substantial". How po-faced can you get?