• cirne (unregistered) in reply to Stephen

    +1. I ran into the same thing Alex did with my colocated server... ONCE. Thankfully I had a sideband channel to the power supply, so I could force a restart remotely. Of course, I had just tweaked the network configuration, so I spend the next couple minutes praying that the server responded to ping again, which luckily it did, or it would have been a two-hour drive for me to visit the server in person.

    TRWTF in this series of screwups wasn't him issuing the command over an unprotected remote connection, it wasn't the server mixup, it was him doing any sort of server reconfiguration task during business hours, because you never know, ESPECIALLY if you're new but even if you're a veteran, what will go wrong. Susan told him not to do things like that during business hours, he didn't listen, and that, not any of the other things that went wrong, is why he should have been fired or at least put on probation.

  • marke (unregistered) in reply to Steve A

    you can fix this you know- even remotely

    you can connect to the registry of the remote computer without RDC sometimes.

    try a variation on http://tinyurl.com/rdcremote

  • tom103 (unregistered)

    The real WTF is the security protocol of that hosting company...

  • (cs) in reply to Sock Puppet 5

    Love it! :)

    Alex isn't even remotely incompetent.

  • (cs) in reply to Sock Puppet 5
    Sock Puppet 5:
    Told from another narrative, Alex could actually be the hero.
    Sock Puppet 5:
    Only one year out of college, Alex, had a huge dilemma. The snowstorm of the century had turned the streets of Massachusetts into a frozen, snow-covered wasteland, and he was frantically working from home as bug reports rolled in. His supervisor was blissfully enjoying her vacation in warmer climes and could not be reached as the heavily-strained network was dropping packets left and right. He knew that restarting the network should not be done during normal business hours, but there was nothing else he knew to try. So, from a remote SSH session he issued the following command:

    /etc/init.d/network stop ; /etc/init.d/network start

    His palm made a beeline for his face as the vanishing SSH window made him realize what had just happened. He paged through his notes like a madman to find the network hosting company to get them to start the network.

    It got worse. The root password was being rejected. He gave it again, paying careful attention to give the proper capitalization and symbols in the 29-digit password. Still no luck. Breathing death threats, he asked the web hosting company to restart the computer and try again. Invalid username/password.

    Panic began to set in, and Alex realized that the fate of the company rested in his hands. Like a superhero, he donned his costume of thermal coveralls, gloves, snowboots, and moose cap. He fired up his Escort, cranked up Wagner on the radio, and burst from his garage like Batman from the batcave. Moments later, he wondered if the person living across the street from Wayne Manor had a sturdier mailbox than the one he just demolished. He left a note and began his Zhivagoan trek to the web hosting headquarters.

    Upon arrival 5-hours later, he examined the machine thoroughly and even reset the password. But files were missing, including the database! Then he noticed that oddly, the server was marked with a different model number than the one he was told the application was running on. A few flagrant obscenities launched at the staff, and the correct server was found. He started the network up with a mix of emotions. Even though he had a momentary brain cramp, which caused all of this, the blame for the five-hour loss of connectivity as well as service outages of the other misfortunate company whose server that had been rebooted (along with his dented fender) lay squarely to blame at the feet of the hosting company. Tired, and relieved that it was now working, he headed home in the dark. During the 5-hour journey back, he resolved never to make that mistake again.

    When Susan finally returned from her vacation, he prepared to start off the morning with his tale of heroic exploits in her absence, along with a laugh about the stupidity and incompetence of the web hosting company.

    Once again, Alex was not in luck.

    I love stories where the "hero" initiates the disaster, then saves the day.

    Nicely written BTW.

  • SysAdWhat? (unregistered)

    The real WTF is that he had to give them the root password over the phone rather than having had an account with appropriate sudo "make me a sandwich" privileges to shutdown and restart the box available to them, right?

  • (cs) in reply to H*a*t*e*r
    H*a*t*e*r:
    boog:
    Fucking go from this fucking site away, you pseudo-"smart" moronic snail
    I did. Managed to get some work done while I was gone. Now I'm back.

    Whee!

  • (cs) in reply to tom103
    tom103:
    The real WTF is the security protocol of that hosting company...
    Indeed. Although if it turned out that the hosting company was working on the wrong machine because Alex originally misread their account/model number or something, I think I would die laughing.
  • Joe Blow (unregistered)

    The website is down.

    http://www.youtube.com/watch?v=W8_Kfjo3VjU

  • (cs) in reply to boog
    boog:
    I love stories where the "hero" initiates the disaster, then saves the day.

    Nicely written BTW.

    In the finest Remy Porter style. But where's the 'corns, dude?

  • (cs)

    Yeah, I love your edit, sock puppet person.

    I admit fully, I once basically made the first half of this mistake - I was trying to update ssh... while connected via ssh. Luckily, I was in the same building as the server I was working on, and it was a school server during the summer. Did take a little while to track down the physical server in the server room, but I had access to it and everything. Felt pretty dumb when I realized what I'd done, though.

    That said, it's not really his mistake (though it is a pretty big wtf) if their web hosting provider gave him access to the wrong machine, and seems an especially big wtf (but still not his) if he managed to successfully change the root password on said machine without having access to the current one. Sounds like the people at that web hosting provider are the ones that need a week of training - in social engineering hacks.

  • (cs) in reply to DCRoss
    DCRoss:
    On the other hand [...]

    On the gripping hand,[...]

    It has been a long while since I have seen someone use this structure properly. Thank you, sir or madam, you have made my day.

  • (cs) in reply to Steve A
    Steve A:
    Reminds me of the time i disabled remote desktop in windows firewall, on the remote server.

    Just kidding, who could be that stupid?

    Just like how I never turned off the NIC of a PC on my home network that was running headless at the back of a packed storage closet.

  • OneMist8k (unregistered)

    My mind is still on the beach, with a visual of Susan.

  • Holy Order of the Zune (unregistered) in reply to RichP
    RichP:
    The story was completely believable until the last line about Susan's mind returning to the beach. Any decent sysadmin's mind would not be on the beach, but on plans for the punishment of the Jr. Admin. Plans involving handcuffs, a ball gag, a blindfold, a switch, a paddle, candles, lit cigarettes, gasoline, high voltage, corrosive acid, a handgun, cinderblocks tied to his ankles, a guillotine, etc.
    ooooooooo.... I would've loved to be Alex right about then. Meeee-OOOWWWW!!!
  • (cs) in reply to OneMist8k
    OneMist8k:
    My mind is still on the beach, with a visual of what I hope Susan looks like.
    FTFY
  • PG4 (unregistered)

    The real WTF is no knowing how to make the changes needed without restarting the whole networking stack.

    The second WTF is all the training classes where you get an official certificate, and they teach, "It's just like a PC restart a service or reboot the machine." rather than fixing the problem.

    BTW, even if you connection doesn't drop try doing a network restart on a clustered server.

  • (cs)

    "So, Susan...time to dust off that résumé?"

  • Obviously Obvious (unregistered)

    Obviously, TRWTF is people named Susan.

  • anon (unregistered) in reply to PG4
    PG4:
    The real WTF is no knowing how to make the changes needed without restarting the whole networking stack.

    The second WTF is all the training classes where you get an official certificate, and they teach, "It's just like a PC restart a service or reboot the machine." rather than fixing the problem.

    BTW, even if you connection doesn't drop try doing a network restart on a clustered server.

    The article makes no mention of what the change was, thus you have absolutely no way of knowing if it could be made without restarting the network stack. And based on the various conclusions you jump to based on a fact you just made up, I'm going to say TRWTF is you.

    And the hosting company. They sound pretty bad.

  • Mojo Monkeyfish (unregistered) in reply to ronpaii

    Yes, of course, because Alex is getting paid less. And he doesn't get vacations, which Susan is acutely aware of every time she takes one.

    Susan makes "rookie errors" (as if you stop f*cking up when you get more experience. what a laugh.) after hours, because she's learned to cover her ass.

    Susan deserves to have a guy hired in above her, who doesn't know sh*t but how to assign her resposibility for every blunder.

  • SuperQ (unregistered) in reply to Remotely fixable

    Yup, remote out of band control/console is required for services that have no failover hardware.

    TRWTF is the lack of escalation path. If you've only got 2 admins, you call the one on vacation when shit is broke.

  • (cs) in reply to RichP
    RichP:
    no situation is so dire that panic cannot make it worse.

    But, as the story proves, no situation is so good that a moment of thoughtless idiocy can't turn it into an occasion for panic.

  • man nohup (unregistered)

    nohup - run a command immune to hangups, with output to a non-tty

  • An Old Hacker (unregistered)

    WTF #1: Hoisting oneself by ones own network connection. WTF #2: Not using screen when doing remote management. WTF #3: Not wondering if the reason that your root password suddenly doesn't work might have to do with what machine it's being tried on. WTF #4: Not being the least bit suspicious when the guy on the other end of the phone can't produce the proper root password. WTF #5: Not being able to tell which physical boxen belong to which clients. SWTF: Not having an escalation procedure in place so that when your incompetent junior does his next FU, you will be contacted to direct resolution. WTHAYTYSB: Not having some sort of watchdog in place to restart the network if it goes done on a business critical server, and a second process to reboot the system if the network restart fails.

    I'm with an earlier commentator: this is a Susan.

  • (cs) in reply to An Old Hacker
    An Old Hacker:
    WTHAYTYSB: Not having some sort of watchdog in place to restart the network if it goes done on a business critical server, and a second process to reboot the system if the network restart fails.
    So if Alex's change to the network config caused the restart to fail, what good is a watchdog or a reboot going to do? Unless, of course, the watchdog is set up to somehow undo his changes.
  • coding god (unregistered) in reply to man nohup
    man nohup:
    nohup - run a command immune to hangups, with output to a non-tty

    You beat me to it. I don't know how many times I had to use that command just to manage over a flaky internet connection.

  • (cs)

    One thing I do love about this article is the many generations of the story.

    1. Alex told the company (he probably had to explain himself), likely omitting some of the most-incriminating details.
    2. Susan heard the story from a co-worker, who probably heard it as a rumor from another co-worker, and so on.
    3. Susan submitted the story to TheDailyWTF, based on what she could piece together.
    4. TheDailyWTF embellished the story and posted it for us.
    5. Each TheDailyWTF reader interpreted the story in his/her own uniquely twisted way, comparing it to his/her own experience.

    The story has been retold so many times, who knows how accurate it really is?

  • Double E (unregistered) in reply to QuesoLoco
    QuesoLoco:
    Susan.

    Seriously, yes he's not qualified, but everyone makes mistakes. She apparently doesn't remember when she was new.

    This is like giving a 15 year old a corvette. She needed to give him tools as she trained him on them. Instead she decided it was more important for her to take a vacation and leave someone not fully trained to manage everything for a week.

    When Susan was "new", she wasn't a senior sysadmin. Her company cheaped out and got what they paid for: someone not competent to be a senior sysadmin.

    Training him might not help: Susan already told Alex not to make network changes during business hours. Did he listen? No. What makes you think he'd do better next time? After this experience, he might. "You need to learn from the mistakes of others; life's too short to make them all yourself", but some people are determined to try.

    And finally, Alex has been their a year and Susan doesn't get to take a week off? How long should she have to wait? Two years? Five? Ten? Until Alex retires?

  • (cs)
    Susan was speechless, not sure which action in this tale of incompetence was the biggest WTF.

    look like it obvius that bigest WTF is susan taking vacations. sysadmins should never go on vacation and if they go, they should cary laptops with conectivity.

  • Obviously Obvious (unregistered) in reply to boog
    boog:
    One thing I do love about this article is the many generations of the story.
    1. Alex told the company (he probably had to explain himself), likely omitting some of the most-incriminating details.
    2. Susan heard the story from a co-worker, who probably heard it as a rumor from another co-worker, and so on.
    3. Susan submitted the story to TheDailyWTF, based on what she could piece together.
    4. TheDailyWTF embellished the story and posted it for us.
    5. Each TheDailyWTF reader interpreted the story in his/her own uniquely twisted way, comparing it to his/her own experience.

    The story has been retold so many times, who knows how accurate it really is?

    Isn't obvious?: 42.37% accurate.

  • (cs) in reply to Obviously Obvious
    Obviously Obvious:
    boog:
    One thing I do love about this article is the many generations of the story.
    1. Alex told the company (he probably had to explain himself), likely omitting some of the most-incriminating details.
    2. Susan heard the story from a co-worker, who probably heard it as a rumor from another co-worker, and so on.
    3. Susan submitted the story to TheDailyWTF, based on what she could piece together.
    4. TheDailyWTF embellished the story and posted it for us.
    5. Each TheDailyWTF reader interpreted the story in his/her own uniquely twisted way, comparing it to his/her own experience.

    The story has been retold so many times, who knows how accurate it really is?

    Isn't obvious?: 42.37% accurate.
    Yes, but which 42.37%?

  • Jeff (unregistered) in reply to Nazi
    Nazi:
    two senior sysadmin’s what?
    You may want to join the 'Society for the 'Suppre's'sion of Apo'strophe Abu'se. Were alway's looking for new member's.
  • eric76 (unregistered) in reply to cirne
    cirne:
    TRWTF in this series of screwups wasn't him issuing the command over an unprotected remote connection, it wasn't the server mixup, it was him doing any sort of server reconfiguration task during business hours, because you never know, ESPECIALLY if you're new but even if you're a veteran, what will go wrong. Susan told him not to do things like that during business hours, he didn't listen, and that, not any of the other things that went wrong, is why he should have been fired or at least put on probation.
    That is precisely right. Do these things during the off hours and be ready to do what you need if the worst happens.
  • (cs) in reply to Jeff
    Jeff:
    Nazi:
    two senior sysadmin’s what?
    You may want to join the 'Society for the 'Suppre's'sion of Apo'strophe Abu'se. Were alway's looking for new member's.
    That sound you heard was the collective Gammar Nazi's' head's exploding.
  • Larry (unregistered) in reply to PedanticCurmudgeon
    PedanticCurmudgeon:
    OneMist8k:
    My mind is still on the beach, with a visual of what I hope Susan looks like.
    FTFY
    Everyone knows that the three females who work in I.T. are all hot.
  • (cs) in reply to Larry
    Larry:
    PedanticCurmudgeon:
    OneMist8k:
    My mind is still on the beach, with a visual of what I hope Susan looks like.
    FTFY
    Everyone knows that the three females who work in I.T. are all hot.
    Really? Not the ones that I've worked with, unless you consider a female that knows more about Star Trek than you, "hot"...

    Disclaimer: I am the image of perfect male physique, BTW...

  • Holy Order of the Zune (unregistered) in reply to C-Octothorpe
    C-Octothorpe:
    Larry:
    PedanticCurmudgeon:
    OneMist8k:
    My mind is still on the beach, with a visual of what I hope Susan looks like.
    FTFY
    Everyone knows that the three females who work in I.T. are all hot.
    Really? Not the ones that I've worked with, unless you consider a female that knows more about Star Trek than you, "hot"...
    I'd like to form an Alliance and cross her Neutral Zone...
    C-Octothorpe:
    Disclaimer: I am the image of perfect male physique, BTW...
    Don't tease me.

    (Why do I always want what's out of reach?)

  • Ha-er (unregistered) in reply to C-Octothorpe
    C-Octothorpe:
    Disclaimer: I am the image of perfect male physique, BTW...
    [image]
  • Obviously Obvious (unregistered) in reply to boog
    boog:
    Obviously Obvious:
    boog:
    Bin-ban-bun, yakirismakiris, blablabla

    The story has been retold so many times, who knows how accurate it really is?

    Isn't obvious?: 42.37% accurate.
    Yes, but which 42.37%?
    Nice question. Let me redo my calculations.

  • geoffrey (unregistered)

    If Susan had properly trained Alex when he started with the company, this would have never happened. This highlights the problem of employees hoarding knowledge in order to secure their own standing, at the expense of their colleagues.

  • Obviously Obvious (unregistered) in reply to Ha-er
    Ha-er:
    C-Octothorpe:
    Disclaimer: I am the image of perfect male physique, BTW...

    [image]

    Ohh, I knew the name C-Octothorpe sounded familiar. How are you? I see you finally grew some hair, nice for you.

  • (cs) in reply to Ha-er
    Ha-er:
    C-Octothorpe:
    Disclaimer: I am the image of perfect male physique, BTW...

    <snip hilarious image of geek>

    WHERE DID YOU FIND THAT PICTURE!!?@?!1!

  • (cs)

    The REAL WTF is that I just logged into one of my test machines to try it...

    /etc/init.d/networking stop ; /etc/init.d/networking start

    It worked just fine, didn't even kill my ssh session.

    root@u64test:~# /etc/init.d/networking stop ; /etc/init.d/networking start * Deconfiguring network interfaces... There is already a pid file /var/run/dhclient.eth0.pid with pid 17435 killed old client process, removed PID file Internet Systems Consortium DHCP Client V3.1.3 Copyright 2004-2009 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/

    Listening on LPF/eth0/00:1c:42:a7:f1:d9 Sending on LPF/eth0/00:1c:42:a7:f1:d9 Sending on Socket/fallback DHCPRELEASE on eth0 to 192.168.2.1 port 67 [ OK ] Rather than invoking init scripts through /etc/init.d, use the service(8) utility, e.g. service networking start

    Since the script you are attempting to invoke has been converted to an Upstart job, you may also use the start(8) utility, e.g. start networking networking stop/waiting root@u64test:~#

  • Bob (unregistered) in reply to Larry
    Larry:
    PedanticCurmudgeon:
    OneMist8k:
    My mind is still on the beach, with a visual of what I hope Susan looks like.
    FTFY
    Everyone knows that the three females who work in I.T. are all hot.

    I had a son who was three females who work in IT, and were all hot, and I can assure you it was no laughing matter.

  • Durn (unregistered) in reply to boog
    boog:
    /etc/init.d/network stop ; /etc/init.d/network start
    WTF did he think would happen?
    I think you're being a little harsh. While an experienced person might realise what's going on, I can well believe almost all of the "fresh out of college" peeps I've worked with would naively make similar mistakes.

    There is always a point between not knowing and knowing where you think you know (but usually don't yet quite know). This is always the most dangerous place to be.

  • Jim (unregistered) in reply to ronpaii
    ronpaii:
    So Alex makes a rooky error. The hosting company shuts down the wrong server causing a 3 hour outage for your company and another. Alex drives to the site, finds the error and fixes it. And you put all this on Alex?
    Maybe you haven['t been here long. Everything is Alex's fault....

    Even Akismet says so...

  • Oogie Boogie (unregistered) in reply to rmeister0
    rmeister0:
    Speaking as a former manager, the problem is actually in the beginning of the story: senior management refused to pay the money it took to hire someone experienced for the role.

    And yes, the hosting company is thrown into serious doubt.

    I call shenanigans. A real manager would never see an issue with management here.

    Oh, unless that's why "former", right?

  • Akismet (unregistered) in reply to Jim
    Jim:
    Maybe you haven['t been here long. Everything is Alex's fault....

    Even Akismet says so...

    It's true. It is.

  • Decius (unregistered) in reply to SysAdWhat?
    SysAdWhat?:
    The real WTF is that he had to give them the root password over the phone rather than having had an account with appropriate sudo "make me a sandwich" privileges to shutdown and restart the box available to them, right?

    Everything else is forgivable. Secure the system by changing the root password and not giving it to this person again. Also, revoke his physical access privileges, or he will hack more servers.

Leave a comment on “Remotely Incompetent”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article