• (cs)

    Another very well written story... so sad it really happened though :)

  • (cs)

    Oh my god. I can feel his dread. The president of the firm brought the server up by himself? That's the real WTF

  • (cs)

    On my first consulting job, I did the same thing. rm * -R from root I was never blamed although I tried to admit my guilt.

  • (cs)

    Nice story. I think a better title would have been "/r♥??& can't be an important file"

  • Rich (unregistered)

    Tip: don't run shit as root if you're really really tired.  Possible additional WTF: How did he get root on a production system if he didn't understand UNIX?

     

    captcha: tps.... go Initech
     

  • GTT (unregistered)

    Someone accidentally let the real name of the company slip :O

  • (cs) in reply to GTT

    Done the FrankenUnix thing - accidentally moved everything from /bin to /usr/bin.  Luckily I was able to link /bin back to /usr/bin w/o any bash commands.

  • Anonymous (unregistered) in reply to PackRat

    PackRat:
    On my first consulting job, I did the same thing. rm * -R from root I was never blamed although I tried to admit my guilt.

    The rm never ran.  The only command that ran would have been the various "mv" commands moving "respectably named files" into the temp dir.  That's why the system was recoverable in the first place - no files were ever deleted, just moved.

    Eventually he started moving important files (presumably like /bin itself) and things started going wrong.  Fortunately since he ALSO moved rm, he didn't have to worry about accidentally running "rm *" on the root path.

    I've accidentally moved /bin, it's not pretty.  (It's amazing what you can do with shell built-ins.  "echo" can work quite nicely for "ls" in a pinch.)  Once you relocate /bin/cp you can start rebuilding the system.
     

  • (cs) in reply to Anonymous

    Silly question, but even if /bin was moved, couldn't you use shell builtins to export a new PATH and continue as normal? I doubt any programs actually have /bin hardcoded into them.

  • Zombie (unregistered) in reply to Rich
    Anonymous:

    Tip: don't run shit as root if you're really really tired.  Possible additional WTF: How did he get root on a production system if he didn't understand UNIX?

     

    captcha: tps.... go Initech
     

    +infinity

     

    That is my number one rule.

     

    captcha: bedtime

    How appropriate...
     

  • (cs)

    Ah, yes... Move all non-gobbledygook files from the root directory to a temp directory.... even the shell interpreter and device nodes... DAHER.... Umm.... yeah.... lol.... How does someone think that it WOULD work?

    Well, ok... you can do it if you are REALLY careful lol...

    Oops.  Uh, it must have been using those files...

    I nominate for WTF quote of the year roflmao.... 

  • Anonymous (unregistered) in reply to bugmenot

    bugmenot:
    Silly question, but even if /bin was moved, couldn't you use shell builtins to export a new PATH and continue as normal? I doubt any programs actually have /bin hardcoded into them.

    #!/bin/sh

    echo "More than you\'d think."

    You could do that to start to recover the system, but you'll need to get the important commands back into /bin.  If /bin/sh (or more likely, /bin/bash, /bin/ksh, /bin/csh, or whatever) has been moved out of /bin, no one can log in anymore.  /etc/passwd refers to shells by direct path.

     

  • rycamor (unregistered)

    "FrankenUnix"

    That alone got a prolonged laugh out of me.
     

  • SwordfishBob (unregistered)

    Which kind of gobbledygook was it?  If it was the result of a noisey phone line, then it's just in the display, and typing 'ls' twice in a row gives different results.  Easy to see it's nothing on the actual system.
    If it was corruption of the file system, then arguably it was all set for failure anyway, though he really shouldn't have tried to fix it all alone, at night, from another state.  (oh yeah, just remembered I do interstate ad-hoc fixes at night myself sometimes.. hm.. )
    If it was just the ERP storing itself in "creative" ways, then it shouldn't have looked quite like the sample screen in the article...

    btw, what's with Symantec advertising "enterprise security software" on a site that equates "enterprise" with WTFery?

  • (cs)

    Ouch.  I did that once on a windows system - "hey lets clean up all temp files, etc."

    whammo "The dynamic link library <dll name> could not be found" error messages from windows when the recursive file deletion routine rolled through c:\windows\system

    <crying> was a luxury at the time as it was a work pc

  • Anonymous (unregistered)

    Some time ago I did something similar.  I was trying to move /var to a new drive.  I mounted the drive to a temp directory, copied the directory, unmounted, and finally remounted /var to the new drive.

     I forgot to preserve permissions.
     

  • orbis tertius (unregistered)

    it may not have been standard in those bygone days.

  • krisztian pinter (unregistered)

    well, to give a comparison for windows and unix.

    i once was asked to clear a windows computer from stuff that is not legal as some kind of audit was planned soon. as things were "hand-installed", the standard uninstall didn't work, so i deleted files, then located info in the registry, and deleted it too. i wanted to delete the HKLM/software/borland key. i'm rather quick with the keyb, so up up up up del enter fit well in a second. the problem is that it was one up more then needed, and borland was the first key. so i deletede the entire HKLM/software.

    now the comparison. it was not really a question how to get the system back to work. it was a complete reinstall of opsys and all software from scratch. at least the audit found no illegal stuff.

  • Not_giving_my_name_for_this_reply (unregistered)

    A long, long time ago in a workplace far away, I was a neophyte Unix user with root access to a variety of machines. We were IT insurgents! 98% of engineering was shackled to the CDC cyber, the enlightened ones had access to the super powerful DEC VMS 8800 (or something like that). Oh but we were beyond that: SGI graphics workstations! DEC Ultrix machines! IBM RS 6000!

    Within two weeks of being granted root access, I totally, hopelessly, unrecoverably hosed the filesystem on one of our DEC ULTRIX workstations. Of course it was at the end of a 16 hr day. I was tired and inexperienced. It was terrible.

     The next morning, I sheepishly confessed to my boss what I had done. His only response was: "That's OK. If Brad (the primary user, developer of a software suite of critical national importance)  doesn't have a backup, he's more of a dumb-ass than I thought.
     

     

     

  • anony-mouse (unregistered) in reply to batasrki

    batasrki:

    The president of the firm brought the server up by himself? That's the real WTF

     

    QFT 

  • (cs) in reply to GoatCheez
    GoatCheez:

    Ah, yes... Move all non-gobbledygook files from the root directory to a temp directory.... even the shell interpreter and device nodes... DAHER.... Umm.... yeah.... lol.... How does someone think that it WOULD work?

    Well, ok... you can do it if you are REALLY careful lol...

    Oops.  Uh, it must have been using those files...

    I nominate for WTF quote of the year roflmao.... 

    I second that motion.

  • rp (unregistered) in reply to bugmenot

    I once hosed a Solaris box in the same way.  The problem is with /lib rather than /bin.
    E.g. on my Linux prompt I see:

    % ldd /bin/sh
    linux-gate.so.1 => (0x0026f000)
    libtermcap.so.2 => /lib/libtermcap.so.2 (0x004e2000)
    libdl.so.2 => /lib/libdl.so.2 (0x003e0000)
    libc.so.6 => /lib/libc.so.6 (0x0028e000)
    /lib/ld-linux.so.2 (0x00270000)
     
    (For you Windows people: an .so. file is a .dll, and ldd lists the ones an executable depends on.
    These days, pretty much every executable depends on at least /lib/libc.so.6 so you only have to rename that file in order to to completely break your system.  In running shells, only the built-in commands will still work, so you can still do echo * to list all files in a directory, but there isn't usually a builtin for file renaming.

    So what you do is reboot from another medium (floppy or something) and repair it that way.  This is standard sysadmin procedure.  I agree with a previous poster, the real WTF is to grant someone like Paul, who has no idea what he's doing let alone how to fix it, root access to a business critical Unix system.

  • Anon (unregistered)

    Ahh, memories: "Oops.  Does anyone have a copy of the database?"

  • Martin (unregistered)

    I once had a friend call me and say, "Uhm. Can you help me? I just did: chmod 666 /bin/* ".

     

  • (cs) in reply to krisztian pinter

    Anonymous:
    well, to give a comparison for windows and unix.

    i once was asked to clear a windows computer from stuff that is not legal as some kind of audit was planned soon. as things were "hand-installed", the standard uninstall didn't work, so i deleted files, then located info in the registry, and deleted it too. i wanted to delete the HKLM/software/borland key. i'm rather quick with the keyb, so up up up up del enter fit well in a second. the problem is that it was one up more then needed, and borland was the first key. so i deletede the entire HKLM/software.

    now the comparison. it was not really a question how to get the system back to work. it was a complete reinstall of opsys and all software from scratch. at least the audit found no illegal stuff.

    Similar to that, I've done my share of "whoops".  I wrote the install/uninstall of our software and we had it set up to (stupidly) recursively delete the install folder when uninstalling.  Of course you can imagine what happens in that directory is "\".  You'd be amazed how quickly you can delete files on a windows machine when you aren't waiting for that stupid progress bar in Windows Explorer.  Luckily we only had to rebuild two developer machines.  I think the next day we all starting using VMWare for doing testing :P
     

  • (cs) in reply to rp

    I was once trying to copy everything in root to a new partition, but I had the old copy level problems, copying the dir rather than it's contents. So I Ctrl+C'd the copy, then did rm -fr * ... on the original root! Hit Ctrl+C pretty quickly but not quickly enough - /bin and /etc were gone. And of course I had no backup, so it was reinstall time. Well, I did have a backup - a month-old one. Just glad it didn't hit /home, was working from Knoppix at the time and /home wasn't mounted.

    I still haven't learnt to take regular backups. Failed sudo apt-get dist-upgrade from dapper to edgy meant another reinstall. edgy being full of issues meant yet another, back to dapper.

  • Dazed (unregistered) in reply to SwordfishBob
    Anonymous:
    Which kind of gobbledygook was it?  If it was the result of a noisey phone line, then it's just in the display, and typing 'ls' twice in a row gives different results.  Easy to see it's nothing on the actual system.
    If it was corruption of the file system, then arguably it was all set for failure anyway ...

    That was my thought too. I've met gobbledygook twice, and both times it signalled a well and truly Foxtroted file system. Is there a relatively benign form of gobbledygook that afflicts Unix? And if so, how do you tidy it up?

    Having said that, moving * from root is a bit like administering sulphuric acid to a patient instead of salicylic acid ...

  • (cs) in reply to PackRat

    And this is why oyu don't run as root. The rm -rf . trick is also a great way of killing your desktop... Time to get the old Ubuntu Live CD out again...

  • (cs) in reply to clem

    clem:
    Another very well written story... so sad it really happened though :)

    I know that feeling of dread. Luckily I've never done anything that stupid at my real job. Of course, my real job is at a company where people like me would never get a root password on *any* server, let alone a production server.

     On my own computer at home I was once trying to make things extra-secure. I thought to myself -- "self, wouldn't it be more secure if only root could list "/". So I did "sudo chmod go-x /". I did "ls -l" immediately after to check my work. DOH! Completely hosed my system. Couldn't run anything. Then I made the mistake of pushing the hard reset button. The worst part is that I didn't have any boot disks (lost the OS install a long time ago) and my keyboard was 3rd party (the computer was a mac) so that I couldn't even boot into single user mode or boot from my external drive. It took me about 12 hours to download (illegally, mind you) enough of the OS from bittorrent sites to piece together a boot disk. Even then it took many hours of fiddling around with the case open -- plugging and unplugging the root volume to control which disc it booted from.

     Another time, I was working on a friends website, and I cd'ed to his html directory...was trying to remove all of the temp files I had put in there, and I did this: "rm -f temp *". Notice the space between "temp" and "*". Now I know why some rm's have a safe mode built in. One little typo and I wiped out his entire HTML directory. I spent several hours dreading waiting for his return phone call. In the end it only took him a few minutes to restore from the development server, but its still not a good feeling to admit you committed a colossal WTF.

     Still, better to do this on your own (or your friends server) than on a client's server.

  • (cs)

    One day I needed to use my desktop computer in the conference room for a demo (all the software wasn't installed on the laptop).  So I carried my desktop (mini-tower) computer into the conference room, plugged in the projector, plugged the desktop into the power strip (on the table), plugged the power strip into the wall, turned on the computer, watched Windows start to boot -- and accidentally dropped a book on the power strip switch.

    Windows writes a lot of things into the registry when it boots, and if the power goes out just as the disk is writing part of the registry ....

    Fortunately I had an earlier "fake" demo on the laptop that I could show, and fortunately it took only the rest of the afternoon to get an earlier version of the registry using the recovery console, and fortunately that earlier version seemed to work OK.

     

  • bruzie (unregistered) in reply to krisztian pinter

    Anonymous:
    well, to give a comparison for windows and unix.

    i once was asked to clear a windows computer from stuff that is not legal as some kind of audit was planned soon. as things were "hand-installed", the standard uninstall didn't work, so i deleted files, then located info in the registry, and deleted it too. i wanted to delete the HKLM/software/borland key. i'm rather quick with the keyb, so up up up up del enter fit well in a second. the problem is that it was one up more then needed, and borland was the first key. so i deletede the entire HKLM/software.

    now the comparison. it was not really a question how to get the system back to work. it was a complete reinstall of opsys and all software from scratch. at least the audit found no illegal stuff.

    You could have rebooted with a rescue disk, deleted the registry data file, and then rebooted. Windows would then have restored the registry from the backup. At least that's the theory - I know it works win win9x.

    But as you say, it's probably easier to hose the OS rather then trying to get rid of the questionable stuff for an audit. 

  • (cs) in reply to rp

    Anonymous:

    So what you do is reboot from another medium (floppy or something) and repair it that way. 

    There are probably wierd Unix variants where this doesn't quite work so easily.  New fangled boot floppies and CDs are great, but if your install media is a 9 track tape and you don't have enough memory for a RAM based filesystem...  I probably would have used the maintenance shell to back up important configuration files and then reinstalled.  Not sure why they couldn't just "mv" the files back from the maintenance shell.  But trying to do things logically when you're panicked and talking to a tech over the phone is not so easy.

  • Anonymous (unregistered)

    Ha! Stories about your own WTFs are the best kind! As long as you're smart enough to realize how stupid you were...

  • Thanks, but no thanks. (unregistered) in reply to orbis tertius

    Ah yes. I wanted to clean up a directory and asked how to do a deletion with confirmation since I wanted to get rid of most, but not all, of the files. The resident guru said "rm * -i". Luckily that Unix implementation didn't know what "-i" meant. Rather than complain, it just went with the part it did understand. I've learned to appreciate RSX and VMS.

  • (cs)

    Early on after developers got their own Unix workstations (SPARCStation 1s) instead of sharing a MicroVAX between 8 users (which is a WTF of its own), I was having a bit of problem with a lack of disk space on mine.  I discovered this huge file in /.  I did a find/grep and couldn't find any reference to it, so I decided that nobody was using it.

     And that's the day when I removed /vmunix.

    Oops.

     

  • maht (unregistered)

    something like : 

    pg_dumpall -c | bzip2 | ssh othercomp '> 100000_customer_records.sql.bz2'

    format old_server's drive & reinstall BSD

    cat 100000_customer_records.sql.bz2 | ssh old_server '| bunzip2 | pgsql'

    bunzip2: (stdin) is not a bzip2 file.

     
    OMG

    turns out the NIC was faulty on old_server and corrupted the bz2 on the way out 

    find only other backup is a month old

    phone boss

     

  • (cs)

    I haven't worked with any flavor of unix for some 10 years. My skillset included creating, removing and listing the contents of directories and not much more and even *I* cringed when I saw he discovered gook in the root directory. I knew he was gonna do it! I knew!

     

  • woah! (unregistered)

    Good ol IT/life lessons.

    I cant think of any major IT wtf I've pulled. I think I learned all of my lessons from the machining industry before I went to college. I once rammed CNC mill tool head in to a part. The machine head was moving 100 inches per minute. The machine had a +/- .0005 tolerance and 40 ft X axis. It moved a 20 ton fixture 3/4 of an inch. I'm lucky I'm not dead. Took us two days to get that sucker square again.

    I went dirt bike riding for the first time by myself, and decided to ignore my dad's advice to walk the bike across city streets. I looked left, looked right, and dumped the clutch and pulled the throttle. Moments later I woked up laying in the middle of the street. My bike was about 50 feet from me and I was laying next to a totalled car. By the grace of god I lived to learn my lesson. He (for some reason) likes me so much I was able to get up and walk away.

    I saw some coworkers do some pretty funny stuff. One added a feature where if a record that was included in a user configurable report was deleted, it sent a friendly email to let them know the point no longer existed. Every minute. I think we sent like 30,000 emails in ten minutes.

    Another one updated all of the email addresses in a 10,000 record user table with his email address on a production server. He denied it, said he didnt know how it happened.

    CAPTCHA: error

     

  • John (unregistered) in reply to ptomblin
    ptomblin:

    ;And that's the day when I removed /vmunix.

    Oops.

     

    Yup. And I guess the original WTF happened because of moving the kernel, not because of moving /bin or /lib. The file is used, even after the kernel was booted. I learned that the hard way a long time ago.

    When was young I found a similar large file with a similar name on HP-UX. A binary which wasn't stripped. Stripping was cool and easy. And that was the day I stripped the HP-UX kernel. The running kernel didn't took it lightly. And rebooting didn't help. Booting from a QIC tape was the name of the game.

  • (cs)

    "Real WTF" is that, it could be deleted with small perl script, or quick c hack program. Just list files and filter all with >50% of non-printable chars. Then delete list.

    Big question is, how they were created in the first place? Did they really get noise on line, that looked like "mkdir ðþêóã¬?whatever" ? I'd say the chances were almost equal to "rm -r /", so that wasn't probably this problem. So how could they be created?

    PS. Almost like:

    update_script upgrade ssh
    pkill sshConnection timeout

    Ooooops....

  • Rich (unregistered) in reply to maht
    Anonymous:

    something like : 

    pg_dumpall -c | bzip2 | ssh othercomp '> 100000_customer_records.sql.bz2'

    format old_server's drive & reinstall BSD

    cat 100000_customer_records.sql.bz2 | ssh old_server '| bunzip2 | pgsql'

    bunzip2: (stdin) is not a bzip2 file.

     
    OMG

    turns out the NIC was faulty on old_server and corrupted the bz2 on the way out 

    find only other backup is a month old

    phone boss

     

    If you're piping through SSH, I can see truncation or issues if the escape character is piped through but I don't think the SSH protocol would allow for simple corruption just because of a dodgy NIC.

    From the man page: 

    The command is run on
              a pty if the client requests a pty; otherwise it is run
              without a tty.  If an 8-bit clean channel is  required,
              one must not request a pty or should specify no-pty.

     

    Rich
  • Renan renan_s2 (unregistered)

    Under Windows (FAT32; never seen that happen with a NTFS system; I have a dual-boot) you could get this type of gibberish files with a corrupted disk. Sometimes, Scandisk is able to fix it, but not always.

     But returning to the subject, a similar story, not involving data loss (thankfully!):

     
    I once decided to compile my own Linux kernel. 'make menuconfig', 'make bzImage', 'make modules', 'make modules_install', copy the new kernel to /boot, edit /etc/lilo.conf (this was under Slackware 10.1, about a year ago), rebooted. Kernel panic! Cannot mount root filesystem blah blah blah.

     
    I checked everything and I had forgotten to enable ReiserFS support on the Linux kernel config (make menuconfig).

     
    Yikes! I had to reboot from a CD, restore the old kernel and rewrite LILO to the bootsector.

     

            
     

  • AndrewG (unregistered) in reply to Martin
    Anonymous:

    I once had a friend call me and say, "Uhm. Can you help me? I just did: chmod 666 /bin/* ".

    Brillant! I love it. So much more creative than the standard "rm -rf /".
    And just imagine if its one of those unix systems where /bin and /usr/bin are linked, and the c compiler, perl, and so on are all in the same directory. Ouch.
     

    Captcha: paula 

  • (cs)

    Rule number 1 when working on a client's server... have a backup.

    So I needed to copy a live data.dbf and data.idx files to a backup folder.

    What I meant to do this: cp data.* backup

    What I ended up doing is this: cp data.*

    Unix kindly expanded my command to the equivilent of: cp data.idx data.dbf

    and made a copy of the index, overwriting the database.  My boss says "well, didn't you make a backup?"  I said "I was TRYING to make the backup!"

  • (cs)

    Urrrgh. Everyone who's ever going to use *nix at all for any purpose, especially as root, should add "Unix Horror Stories" to the list of mandatory reading. (The biggest lesson it can teach is extremely relevant to this story: Be Damn Careful With rm(1).) I read this a couple of years before I touched Linux for the first time, and I suppose that's why I've never done stuff like this. =)

    This entry in particular was among the classic stories it had. Heroic tale of that shall inspire us when your well-meaning friend happens to fry everything while doing crazy stuff as root...

  • Rich (unregistered) in reply to viraptor
    viraptor:

    Big question is, how they were created in the first place? Did they really get noise on line, that looked like "mkdir ðþêóã¬?whatever" ? I'd say the chances were almost equal to "rm -r /", so that wasn't probably this problem. So how could they be created?

    Some terminal programs allow you to upload files. I suppose that there might be one that could recognize line noise as "upload this file with this filename, some number of bytes to follow). Though I have never known one be this fragile (I have met with kermit and zmodem).

     

    I guess if things were really bad, he might be halfway through something like "cp filename1 filename2" when halfway through typing filename2, the linenoise pumps a load of crap in including a nice friendly ^M 

    PPP was a godsend really. Even for those using shells.

     

    Rich 

  • Dark (unregistered)

    Suppose you're innocently using your fancy unix system. Some noise occurs on the modem line you're using; this happens, and usually it only messes up your display. But this noise can also be received as input to the command shell, in two ways: either it triggers a terminal escape code that generates input, or it occurs on the sending side of the line in the first place.

    Now suppose there's a ">" character in the noise. The command shell, obviously, interprets this as output redirection, with the name of the output file following it. It then creates this output file before doing anything else -- before, for example, checking whether the start of the noise was actually a valid command name.

    The user just notices a burst of noise, thinks "drat" and continues. But since (obviously) they were always logged in as root, each such event left a weirdly named 0-length file in whatever directory they were in.

    It's exciting, but not as exciting as the "paste entire terminal history as input" mouse click combination that was easily triggered by accident on certain SunOS systems :)

  • Juliio C. Oliveira (unregistered)

    man, I remenber the day a accidentally issued a halt command on a remote production server... 30 minutes to get someone to press the f*** power button...
     

  • Rich (unregistered) in reply to Rich
    Anonymous:

    If you're piping through SSH, I can see truncation or issues if the escape character is piped through but I don't think the SSH protocol would allow for simple corruption just because of a dodgy NIC.

    From the man page: 

    The command is run on
              a pty if the client requests a pty; otherwise it is run
              without a tty.  If an 8-bit clean channel is  required,
              one must not request a pty or should specify no-pty.

     

    Rich

    Never mind, I see SSH will not allocate a TTY when running a command. I still don't see that ssh would corrupt data. Truncate it maybe.

     

    Rich 

  • (cs) in reply to WWWWolf
    WWWWolf:

    Urrrgh. Everyone who's ever going to use *nix at all for any purpose, especially as root, should add "Unix Horror Stories" to the list of mandatory reading. (The biggest lesson it can teach is extremely relevant to this story: Be Damn Careful With rm(1).) I read this a couple of years before I touched Linux for the first time, and I suppose that's why I've never done stuff like this. =)


    I'm a little paranoid on this, but I never use wildcards when deleting files as root.

Leave a comment on “It Must Have Been Using Those Files”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article