• Haters Gonna Hate (unregistered)

    first.

    eros, i am not a robot.

  • Steve A (unregistered)

    Reminds me of the time i disabled remote desktop in windows firewall, on the remote server.

    Just kidding, who could be that stupid?

  • Baboon (unregistered)

    Haha love it ... Frist?

  • Eddy (unregistered)

    Fr1st?

  • Nagesh (unregistered)

    Susan is matterhorn, not remeber being beginer one time herself.

  • (cs)

    Frixth!

    And no Askimet, this is not spam

  • (cs)

    I don't know why they do it, but I know of a NUMBER of folks who have disabled the network connection on their Windows Home Servers running headless on an HP MediaSmart Server ... which has no keyboard/mouse/video ports.

  • (cs)
    /etc/init.d/network stop ; /etc/init.d/network start
    WTF did he think would happen?
    He gave the hosting company the root password over the phone, but it was rejected...had them reboot the server and try to login again...boot directly into the shell and reset the root password...the hosting company had him working on another company’s server.
    Hacking into your competitors' servers is easy when the hosting company helps you out. You'd think when the guy didn't know the root password, they'd have stopped and wondered, "Is this the right server? Is this guy legit?"

    Yay for corporate espionage!

  • Medinoc (unregistered)

    Wait... How could the hosting company work on the wrong server?

    This Alex give them a WTF client number, or did they pile a blunder of their own on his?

  • (cs)

    Alex screwed up, yes, but how is it that the hosting company allowed him to start screwing around on someone else's machine - and even screwed around on someone else's machine for him? Susan's first task should be to find a new hosting company, no?

  • QuesoLoco (unregistered)

    Susan.

    Seriously, yes he's not qualified, but everyone makes mistakes. She apparently doesn't remember when she was new.

    This is like giving a 15 year old a corvette. She needed to give him tools as she trained him on them. Instead she decided it was more important for her to take a vacation and leave someone not fully trained to manage everything for a week.

  • ronpaii (unregistered)

    So Alex makes a rooky error. The hosting company shuts down the wrong server causing a 3 hour outage for your company and another. Alex drives to the site, finds the error and fixes it. And you put all this on Alex?

  • Medinoc (unregistered) in reply to boog
    boog:
    /etc/init.d/network stop ; /etc/init.d/network start
    WTF did he think would happen?
    Well, he probably thought killing the connection wouldn't kill his session in the middle of the shell executing a batch.

    Under Windows Terminal Server, he'd have been right.

  • (cs) in reply to QuesoLoco
    QuesoLoco:
    Susan.

    Seriously, yes he's not qualified, but everyone makes mistakes. She apparently doesn't remember when she was new.

    This is like giving a 15 year old a corvette. She needed to give him tools as she trained him on them. Instead she decided it was more important for her to take a vacation and leave someone not fully trained to manage everything for a week.

    Oh boy, another troll saying the submitter is TRWTF. I love these.

  • (cs) in reply to QuesoLoco
    QuesoLoco:
    Susan.

    Seriously, yes he's not qualified, but everyone makes mistakes. She apparently doesn't remember when she was new.

    This is like giving a 15 year old a corvette. She needed to give him tools as she trained him on them. Instead she decided it was more important for her to take a vacation and leave someone not fully trained to manage everything for a week.

    Yes, cause after almost a year, Susan's still not allowed to take a week off due to someone who, in that same time, hasn't gotten the hang of basic maintenance yet. The biggest issue here, is that hosting company, though. And the N key on my keyboard.

  • (cs) in reply to boog
    boog:
    QuesoLoco:
    Susan.

    Seriously, yes he's not qualified, but everyone makes mistakes. She apparently doesn't remember when she was new.

    This is like giving a 15 year old a corvette. She needed to give him tools as she trained him on them. Instead she decided it was more important for her to take a vacation and leave someone not fully trained to manage everything for a week.

    Oh boy, another troll saying the submitter is TRWTF. I love these.
    The problem is that there seems to be more every day. I call them TRWTF apologists...

  • adamsb6 (unregistered) in reply to Medinoc

    I've done an /etc/init.d/network restart through an SSH session plenty of times and haven't seen my session get dropped.

  • JT (unregistered)

    Not that I'd restart the network without a better fallback plan, but I firmly believe that one should always, always use GNU screen for the very reason that you won't have unexpected disconnections causing just that sort of problem.

  • Stephen (unregistered) in reply to Medinoc

    This is why you run everything in screen.

  • (cs) 
    /etc/init.d/network stop ; /etc/init.d/network start
    If this had been inside a tmux or screen session, it would have been perfectly fine. The command would run to completion, taking all the nets down, and back up, and then Alex could have logged back in again.

    So the real WTF is running admin commands while not in a protected session. After all, you never know when the net will break.

  • (cs)

    The story was completely believable until the last line about Susan's mind returning to the beach. Any decent sysadmin's mind would not be on the beach, but on plans for the re-education of the Jr. Admin. Plans involving high voltage, corrosive acid, sharp objects, etc.

    Alex has just discovered one of the most important rules of system administration: no situation is so dire that panic cannot make it worse.

  • Guybrush (unregistered) in reply to ronpaii

    Does that mean 'having the characteristics of a smallish crow'?

    Excellent CAPTCHA: "validus"

  • Nazi (unregistered)

    two senior sysadmin’s what?

  • (cs) in reply to Medinoc
    Medinoc:
    boog:
    /etc/init.d/network stop ; /etc/init.d/network start
    WTF did he think would happen?
    Well, he probably thought killing the connection wouldn't kill his session in the middle of the shell executing a batch.

    Under Windows Terminal Server, he'd have been right.

    But he wasn't, so he was wrong.

    Still, even if he was right, he just changed the network configuration, so he should have known there would be an increased possibility that the startup would fail. Or maybe this is Dunning-Kruger in practice?

  • powercf (unregistered) in reply to ronpaii
    ronpaii:
    So Alex makes a rooky error. The hosting company shuts down the wrong server causing a 3 hour outage for your company and another. Alex drives to the site, finds the error and fixes it. And you put all this on Alex?

    +1

    Honest mistake. Not something I've done myself yet, but I can see how it would happen. The good news is that Alex will be very careful in future when doing any network/firewall work remotely

  • (cs)
    he noticed the model on the side of the server did not match the model he had been told the server was running on.

    Cindy Crawford vs. Heidi Klum?

  • rmeister0 (unregistered)

    Speaking as a former manager, the problem is actually in the beginning of the story: senior management refused to pay the money it took to hire someone experienced for the role.

    And yes, the hosting company is thrown into serious doubt.

  • DCRoss (unregistered) in reply to boog
    boog:
    WTF did he think would happen?
    He probably thought that the network would restart.

    That's what happens when I run those two commands together. Even in a remote SSH session. In fact I did it just now and it worked without dropping my session or causing a panic.

    On the other hand my network is set up correctly so this should work. If Alex had just changed his IP address to something like "192168.1.1" then there would be no hope for him.

    On the gripping hand, he did try to do scheduled maintenance on a production server during business hours, and neither he nor the hosting company seemed to know how to access the console remotely, so he earns a keyboard to the back of the head for that.

  • (cs) in reply to RichP
    RichP:
    ...no situation is so dire that panic cannot make it worse.
    Nice. I'm writing that one down.
  • (cs) in reply to RichP
    RichP:
    Any decent sysadmin's mind would not be on the beach, but on plans for the re-education of the Jr. Admin. Plans involving high voltage, corrosive acid, sharp objects, etc.
    hoodaticus? Is that you?

    Did zunesis finally release you from his dungeon?

  • (cs) in reply to DCRoss
    DCRoss:
    boog:
    WTF did he think would happen?
    He probably thought that the network would restart.
    After a "minor change" to the network config? Maybe he needs some lessons in risk assessment.
  • Sysadmin (unregistered)

    Do people really run servers at remote hosting companies without some kind of lights-out management?

    Now, that's a WTF.

  • Robb (unregistered)

    Susan needs to read stuff from Bastard Operator from Hell. I recall one specifically on dealing with new sys admins.

  • Bob (unregistered)

    The command shouldn't have killed his SSH session (assuming the server came back up within the timeout).

    The only clue is, "he had just made a minor change to the network configuration".

    Presumably, the story should actually end, "A few rows down, Alex found his computer, logged in, fixed the typo he'd introduced and restarted the network connection."

  • other (unregistered) in reply to realmerlyn
    realmerlyn:
    /etc/init.d/network stop ; /etc/init.d/network start
    If this had been inside a tmux or screen session, it would have been perfectly fine. The command would run to completion, taking all the nets down, and back up, and then Alex could have logged back in again.

    So the real WTF is running admin commands while not in a protected session. After all, you never know when the net will break.

    I've never worked as a Sysadmin, but doesn't the network service have a restart option? That looks like it would leave the network running, but still kill the SA's login shell.

    /etc/init.d/network restart
  • Alex (no not the one from the story) (unregistered)

    Yay, so he makes an easy mistake. Boo the hosting company screw him over by giving him access to someone elses hosted server (and then when they have completely failed to notice what a fcuk up they are making give him physical access to it WTF????)

  • AP2 (unregistered)

    Restarting the network in a SSH session without screen is an excusable mistake. Messing with the server during work hours, after being told not to, is Very Bad.

    Of course, TRWTF is the hosting company.

  • Herr Otto Flick (unregistered) in reply to Stephen
    Stephen:
    This is why you run everything in tmux.

    FTFY

  • Veritasiness (unregistered)

    TRWTF is definitely the hosting company - seriously, giving him access to the WRONG server?

    Not knowing about the restart command isn't as much a WTF as it is a laughably sad lack of experience.

  • (cs) in reply to RichP
    RichP:
    Alex has just discovered one of the most important rules of system administration: no situation is so dire that panic cannot make it worse.

    We had an event in our data center that caused one of our most critical systems to blink out for less than a minute. This caused Citrix servers to drop hundreds of sessions. When they dropped the sessions, everyone started to reconnect simultaneously and that caused some of the Citrix servers to fall over. A total clusterfsck.

    During the time when we were trying to figure out the problem, I had one of the server engineers (generally a bright guy, but . . .) come up and say "We can't just stand here. Reboot something!"

  • AP2 (unregistered) in reply to other
    other:
    realmerlyn:
    /etc/init.d/network stop ; /etc/init.d/network start
    If this had been inside a tmux or screen session, it would have been perfectly fine. The command would run to completion, taking all the nets down, and back up, and then Alex could have logged back in again.

    So the real WTF is running admin commands while not in a protected session. After all, you never know when the net will break.

    I've never worked as a Sysadmin, but doesn't the network service have a restart option? That looks like it would leave the network running, but still kill the SA's login shell.

    /etc/init.d/network restart

    It does, but at least in Debian it also prints:

    "Running /etc/init.d/networking restart is deprecated because it may not enable some interfaces"

    And if you run it without arguments, it shows:

    "Usage: /etc/init.d/networking {start|stop}"

  • Anonymous (unregistered) in reply to AP2

    Thank you.

    OT: Akismet said this comment was spam. Great.

  • Herr Otto Flick (unregistered) in reply to realmerlyn
    realmerlyn:
    /etc/init.d/network stop ; /etc/init.d/network start
    If this had been inside a tmux or screen session, it would have been perfectly fine. The command would run to completion, taking all the nets down, and back up, and then Alex could have logged back in again.

    So the real WTF is running admin commands while not in a protected session. After all, you never know when the net will break.

    I don't know about play OS like Linux, but if you do that on a real Unix box, tmux or not, it won't drop the session.

    It will drop the session if you fuck up the new network configuration such that either the ifconfig is now bullshit, or unintended change causes nothing to flow through the firewall.

    In those cases, tmux or screen won't help you, you aimed at your foot and pulled the trigger, welcome to UNIX.

  • (cs) in reply to boog

    I dunno about this.. but I know that /etc/init.d/network restart can work fine without even killing your ssh connection, which as far as I know is magic.

    Of course, I've only ever done this when I have physical access when it doesn't work the first time.

  • Bananas (unregistered) in reply to adamsb6
    adamsb6:
    I've done an /etc/init.d/network restart through an SSH session plenty of times and haven't seen my session get dropped.
    You saw 
    /etc/init.d/network stop; /etc/init.d/network start
    and read 
    /etc/init.d/network restart
    You'd better improve your reading skills before your next job interview.
  • (cs) in reply to boog
    boog:
    QuesoLoco:
    Susan.

    Seriously, yes he's not qualified, but everyone makes mistakes. She apparently doesn't remember when she was new.

    This is like giving a 15 year old a corvette. She needed to give him tools as she trained him on them. Instead she decided it was more important for her to take a vacation and leave someone not fully trained to manage everything for a week.

    Oh boy, another troll saying the submitter is TRWTF. I love these.
    Told from another narrative, Alex could actually be the hero.

    Sock Puppet 5:
    Only one year out of college, Alex, had a huge dilemma. The snowstorm of the century had turned the streets of Massachusetts into a frozen, snow-covered wasteland, and he was frantically working from home as bug reports rolled in. His supervisor was blissfully enjoying her vacation in warmer climes and could not be reached as the heavily-strained network was dropping packets left and right. He knew that restarting the network should not be done during normal business hours, but there was nothing else he knew to try. So, from a remote SSH session he issued the following command:

    /etc/init.d/network stop ; /etc/init.d/network start

    His palm made a beeline for his face as the vanishing SSH window made him realize what had just happened. He paged through his notes like a madman to find the network hosting company to get them to start the network.

    It got worse. The root password was being rejected. He gave it again, paying careful attention to give the proper capitalization and symbols in the 29-digit password. Still no luck. Breathing death threats, he asked the web hosting company to restart the computer and try again. Invalid username/password.

    Panic began to set in, and Alex realized that the fate of the company rested in his hands. Like a superhero, he donned his costume of thermal coveralls, gloves, snowboots, and moose cap. He fired up his Escort, cranked up Wagner on the radio, and burst from his garage like Batman from the batcave. Moments later, he wondered if the person living across the street from Wayne Manor had a sturdier mailbox than the one he just demolished. He left a note and began his Zhivagoan trek to the web hosting headquarters.

    Upon arrival 5-hours later, he examined the machine thoroughly and even reset the password. But files were missing, including the database! Then he noticed that oddly, the server was marked with a different model number than the one he was told the application was running on. A few flagrant obscenities launched at the staff, and the correct server was found. He started the network up with a mix of emotions. Even though he had a momentary brain cramp, which caused all of this, the blame for the five-hour loss of connectivity as well as service outages of the other misfortunate company whose server that had been rebooted (along with his dented fender) lay squarely to blame at the feet of the hosting company. Tired, and relieved that it was now working, he headed home in the dark. During the 5-hour journey back, he resolved never to make that mistake again.

    When Susan finally returned from her vacation, he prepared to start off the morning with his tale of heroic exploits in her absence, along with a laugh about the stupidity and incompetence of the web hosting company.

    Once again, Alex was not in luck.

  • H*a*t*e*r (unregistered) in reply to boog
    boog:
    But he wasn't, so he was wrong.

    Still, even if he was right, he just changed the network configuration, so he should have known there would be an increased possibility that the startup would fail. Or maybe this is Dunning-Kruger in practice?

    Fucking go from this fucking site away, you pseudo-"smart" moronic snail

  • Remotely fixable (unregistered)
    After realizing his mistake, Alex panicked and tried a wide variety of tasks to fix a remote computer with no internet, which Susan was told probably involved praying to some deity.
    What?! This is doable. Never heard of IPMI? It even works via a serial interface.
  • (cs) in reply to Sock Puppet 5
    Sock Puppet 5:
    Only one year out of college, Alex, had a huge dilemma. The snowstorm of the century had turned the streets of Massachusetts into a frozen, snow-covered wasteland, and he was frantically working from home as bug reports rolled in. His supervisor was blissfully enjoying her vacation in warmer climes and could not be reached as the heavily-strained network was dropping packets left and right. He knew that restarting the network should not be done during normal business hours, but there was nothing else he knew to try. So, from a remote SSH session he issued the following command:

    /etc/init.d/network stop ; /etc/init.d/network start

    His palm made a beeline for his face as the vanishing SSH window made him realize what had just happened. He paged through his notes like a madman to find the network hosting company to get them to start the network.

    It got worse. The root password was being rejected. He gave it again, paying careful attention to give the proper capitalization and symbols in the 29-digit password. Still no luck. Breathing death threats, he asked the web hosting company to restart the computer and try again. Invalid username/password.

    Panic began to set in, and Alex realized that the fate of the company rested in his hands. Like a superhero, he donned his costume of thermal coveralls, gloves, snowboots, and moose cap. He fired up his Escort, cranked up Wagner on the radio, and burst from his garage like Batman from the batcave. Moments later, he wondered if the person living across the street from Wayne Manor had a sturdier mailbox than the one he just demolished. He left a note and began his Zhivagoan trek to the web hosting headquarters.

    Upon arrival 5-hours later, he examined the machine thoroughly and even reset the password. But files were missing, including the database! Then he noticed that oddly, the server was marked with a different model number than the one he was told the application was running on. A few flagrant obscenities launched at the staff, and the correct server was found. He started the network up with a mix of emotions. Even though he had a momentary brain cramp, which caused all of this, the blame for the five-hour loss of connectivity as well as service outages of the other misfortunate company whose server that had been rebooted (along with his dented fender) lay squarely to blame at the feet of the hosting company. Tired, and relieved that it was now working, he headed home in the dark. During the 5-hour journey back, he resolved never to make that mistake again.

    When Susan finally returned from her vacation, he prepared to start off the morning with his tale of heroic exploits in her absence, along with a laugh about the stupidity and incompetence of the web hosting company.

    Once again, Alex was not in luck.

    +1

    Awesome!

  • (cs) in reply to Nazi
    Nazi:
    two senior sysadmin’s what?
    +1000

Leave a comment on “Remotely Incompetent”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article