• foo (unregistered) in reply to boog
    boog:
    You're missing the point. Peak hours are defined as the hours when the volume of traffic is highest. I don't need data, because I'm relying on the definitions of the words.
    Wow. The article says "after hours", many commenters say "business hours", you just change it to "peak hours" and rely on the definition of "peak". You should go into politics. (Tip: Next time you try this trick, pass a little time between. Then noone might notice.)
    boog:
    foo:
    OMG, I offended the big boog, and now (s)he's shaming me.
    Who said anything about being offended? (Now who's speculating?)
    I never denied speculating. Though based on your reaction it seems a rather safe guess.
    boog:
    foo:
    boog:
    I'm not sure what facts you think I need to prove to you.
    How about any, but I guess that would be asking too much.
    Let me rephrase that - WTF exactly am I supposed to be proving?
    Who said anything about proving anyway? (Hint: you, three quotes above.)

    I asked a question, you responded with speculation, I countered with another speculation, that's all.

    boog:
    foo:
    Many posters blamed Alex for this without ever question whether it was a valid requirement. Just because Susan told him so? Susan's credibility was already cast in doubt. Given the circumstances, doing it by day might have had advantages, such as not having to drive to the hosting company in the middle of the night, possibly just to find some incompetent night-shift there. (Yes, I know, he found the incompetent day-shift, but he couldn't foresee this.)
    That's completely stupid. Worrying about who's on support at the hosting company is hardly a concern when compared to that of losing money.
    No support -> longer downtime -> losing money. How is that hard to get? And don't forget driving through a snowstorm at night.
    boog:
    Peak hours = lots of users = if your site is down you lose money. How are you still not getting that?
    Sure I get that ... that it's speculation. Depends entirely on the kind of web site. Many web sites are used more during leisure time, I guess.

    BTW, I'm not really interested in beating the dead horse, as you rightly called it. It's just that arguing with you is so much fun because you make it so easy -- contradicting yourself, "proving" speculation with more speculation. You don't often get to see that for so long. Please keep up the good work! (And I do apologise for the last paragraphs which actually discuss parts of the story.)

  • (cs) in reply to dna
    dna:
    gess who just remotly restart a server after writing the wrong gateway address ^^

    and i have 10 years of experience, and 3 years in that particular job...

    sometimes, you juste made mistake...

    Yes, But did you do it on a busy production server during peak hours (not during a scheduled downtime) and were you breaking any rules in the SOP your company has outlined for you? Also, were you doing it in such a fashion that you cannot get to (or get reliable hands to) the server (such as during a record blizzard?).

    If so you should be fired for incompetence. If not then mistakes are mistakes and you will learn.

  • (cs) in reply to foo
    foo:
    Wow. The article says "after hours", many commenters say "business hours", you just change it to "peak hours" and rely on the definition of "peak".
    Actually, I said "peak hours" in my first answer to you. So your problem this whole time has been that you didn't like me suggesting that peak hours = office hours? Why didn't you just say that in the first place?
    foo:
    boog:
    foo:
    OMG, I offended the big boog, and now (s)he's shaming me.
    Who said anything about being offended? (Now who's speculating?)
    I never denied speculating. Though based on your reaction it seems a rather safe guess.
    Who said you denied speculating? Really, and you tell me to look it up in a dictionary.
    foo:
    boog:
    foo:
    boog:
    foo:
    OK, so you're not stating facts (since you can't prove them)
    I'm not sure what facts you think I need to prove to you.
    How about any, but I guess that would be asking too much.
    Let me rephrase that - WTF exactly am I supposed to be proving?
    Who said anything about proving anyway? (Hint: you, three quotes above.)
    Wrong again; that was a direct response to you saying I can't. Am I the only one actually following this discussion?
    foo:
    No support -> longer downtime -> losing money. How is that hard to get? And don't forget driving through a snowstorm at night.
    It's not hard to get at all. I'm comparing that risk to the risk of losing business during the day. I've never had to worry about increased downtime at night, and even if I did the impact to the users would still have been minimal.

    That said, I can agree that I'd personally never want to drive through a snowstorm at night for work. If night-maintenance would have required that, I'd have simply rescheduled the maintenance work for another night, one without a snowstorm.

    Also, if the night-shift was really so fucking incompetent, to the point where I'd rather put up with the risks of daytime maintenance (even if those risks include downtime during peak hours), then I'd postpone the maintenance until close to the end of the day-shift, rather than a time of day that would cost a whole day's worth of orders.

    foo:
    It's just that arguing with you is so much fun because you make it so easy -- contradicting yourself, "proving" speculation with more speculation.
    That's cute. Maybe you could point out for me exactly where I've contradicted myself, or when I claimed speculation was proof of speculation. I'm curious to know, so I could be less likely to make the mistake again in future.
  • (cs) in reply to Sock Puppet 5
    Sock Puppet 5:
    Only one year out of college, Alex, had a huge dilemma. The snowstorm of the century had turned the streets of Massachusetts into a frozen, snow-covered wasteland, and he was frantically working from home as bug reports rolled in. His supervisor was blissfully enjoying her vacation in warmer climes and could not be reached as the heavily-strained network was dropping packets left and right. He knew that restarting the network should not be done during normal business hours, but there was nothing else he knew to try. So, from a remote SSH session he issued the following command:

    /etc/init.d/network stop ; /etc/init.d/network start

    His palm made a beeline for his face as the vanishing SSH window made him realize what had just happened. He paged through his notes like a madman to find the network hosting company to get them to start the network.

    It got worse. The root password was being rejected. He gave it again, paying careful attention to give the proper capitalization and symbols in the 29-digit password. Still no luck. Breathing death threats, he asked the web hosting company to restart the computer and try again. Invalid username/password.

    Panic began to set in, and Alex realized that the fate of the company rested in his hands. Like a superhero, he donned his costume of thermal coveralls, gloves, snowboots, and moose cap. He fired up his Escort, cranked up Wagner on the radio, and burst from his garage like Batman from the batcave. Moments later, he wondered if he should have raised the garage door before bursting from his garage... but there was no turning back now.

    Upon arrival 5-hours later, he examined the machine thoroughly and even reset the password. But files were missing, including the database! Then he noticed that oddly, the server was marked with a different model number than the one he was told the application was running on. A few flagrant obscenities launched at the staff, and the correct server was found. He started the network up with a mix of emotions. Even though he had a momentary brain cramp, which caused all of this, the blame for the five-hour loss of connectivity as well as service outages of the other misfortunate company whose server that had been rebooted (along with his dented fender) lay squarely to blame at the feet of the hosting company. Tired, and relieved that it was now working, he headed home in the dark. During the 5-hour journey back, he resolved never to make that mistake again.

    When Susan finally returned from her vacation, he prepared to start off the morning with his tale of heroic exploits in her absence, along with a laugh about the stupidity and incompetence of the web hosting company.

    Once again, Alex was not in luck.

    FTFY

  • (cs) in reply to DCRoss
    DCRoss:
    boog:
    WTF did he think would happen?
    He probably thought that the network would restart.

    That's what happens when I run those two commands together. Even in a remote SSH session. In fact I did it just now and it worked without dropping my session or causing a panic.

    On the other hand my network is set up correctly so this should work. If Alex had just changed his IP address to something like "192168.1.1" then there would be no hope for him.

    On the gripping hand, he did try to do scheduled maintenance on a production server during business hours, and neither he nor the hosting company seemed to know how to access the console remotely, so he earns a keyboard to the back of the head for that.

    RUN!! IT'S A MOTIE!!

  • cappeca (unregistered) in reply to ronpaii
    ronpaii:
    So Alex makes a rooky error. The hosting company shuts down the wrong server causing a 3 hour outage for your company and another. Alex drives to the site, finds the error and fixes it. And you put all this on Alex?

    +1

  • Calvin (unregistered) in reply to boog
    boog:
    /etc/init.d/network stop ; /etc/init.d/network start
    WTF did he think would happen?

    Well, I'd have thought that was OK (not being a Linux admin)

    But, I bet most people have done something similar at least once in their lives. It's one of the things you learn quickly by experience...

    I know I've reconfigured remote servers and made it so I can't log in remotely - that's why I always make sure we have remote KVM support to our servers as well - that way there's two things I have to kill before I get total loss of access...

    IMV, the real WFTs are:

    • the hosting company...
    • his company for not having remote KVM support

    Alex made two mistakes, one which wasn't blindingly obvious to someone who hadn't already made it before... The other mistake (restarting the service during the day) probably wasn't that big in the end - if he'd restarted it in the evening, it would probably have taken much longer to get it fixed.

    The rest was him doing a valiant job trying to fix it, and being hindered by everyone/everything else.

    So, Alex deserves a medal. The main thing he did wrong, you can bet he won't be doing again...

  • Calvin (unregistered) in reply to Socio
    Socio:
    Susan needs only show the boss (the one who collects the amount shown in the company's bottom line) this inequality:

    (Crappy Hosting Company Fees) + (One Day's Orders) > (Good Hosting Company Fees)

    FTFY

  • Michaelangelo (unregistered) in reply to Durn
    Durn:
    boog:
    /etc/init.d/network stop ; /etc/init.d/network start
    WTF did he think would happen?
    I think you're being a little harsh. While an experienced person might realise what's going on, I can well believe almost all of the "fresh out of college" peeps I've worked with would naively make similar mistakes.

    There is always a point between not knowing and knowing where you think you know (but usually don't yet quite know). This is always the most dangerous place to be.

    Also, how would you learn not to do that? Is there a 'things a sysadmin should never do' book?

    I bet most people learn not to do it by doing it, then saying 'oh crap', and having to walk^H^H^H^Hrun madly down to the server room to fix things.

    Alex was just unlucky that his learning experience was at a hosting company hours away, and that his employer was too cheap to get a decent hosting company, remote power switching, or remote KVM.

  • foo (unregistered) in reply to boog
    boog:
    foo:
    Wow. The article says "after hours", many commenters say "business hours", you just change it to "peak hours" and rely on the definition of "peak".
    Actually, I said "peak hours" in my first answer to you.
    Oh, sorry, my fault. You did actually heed my advice and switch terms early and let some time pass so I actually missed it. Congratulations, you're a senior politician.
    boog:
    foo:
    boog:
    foo:
    OMG, I offended the big boog, and now (s)he's shaming me.
    Who said anything about being offended? (Now who's speculating?)
    I never denied speculating. Though based on your reaction it seems a rather safe guess.
    Who said you denied speculating? Really, and you tell me to look it up in a dictionary.
    Who said anyone said I denied speculating?
    boog:
    foo:
    boog:
    foo:
    boog:
    foo:
    OK, so you're not stating facts (since you can't prove them)
    I'm not sure what facts you think I need to prove to you.
    How about any, but I guess that would be asking too much.
    Let me rephrase that - WTF exactly am I supposed to be proving?
    Who said anything about proving anyway? (Hint: you, three quotes above.)
    Wrong again; that was a direct response to you saying I can't. Am I the only one actually following this discussion?
    Ah, that. Well you misunderstood me. I didn't say you need to prove things (since I don't think you can). I just pointed out (though somewhat implicitly -- sorry I assumed you could figure this out yourself) that statements can be grouped into 3 categories: 1. proved => true 2. disproved => false 3. neither proved nor disproved => speculation Since your statements obviously aren't 1. and you denied 3., this logially leaves only 2. by your assertion. (I'm still willing to give you 3., but if you don't want ...)
    boog:
    That said, I can agree that I'd personally never want to drive through a snowstorm at night for work. If night-maintenance would have required that, I'd have simply rescheduled the maintenance work for another night, one without a snowstorm.
    And how do you know rescheduling was possible? (Speculation?) The article says: "[Alex] decided it was important for the web server’s network connection to be restarted." Maybe he was right. (My speculation.)
    boog:
    foo:
    It's just that arguing with you is so much fun because you make it so easy -- contradicting yourself, "proving" speculation with more speculation.
    That's cute. Maybe you could point out for me exactly where I've contradicted myself,
    How about claiming above that you followed the discussion and then asking a question like this:
    boog:
    or when I claimed speculation was proof of speculation.
    You didn't claim it was, you just behave like it was. As for when, all the time. Your whole argument rests on the claim (i.e., speculation) that there are more customers during business hours. You change words, rephrase it and restate your claim at infinitum, but in the end it's just a circular argument which proves nothing.
  • vik (unregistered)

    A female sys admin ??????

  • Arancaytar (unregistered)
    Alex looked down, about ready to run away and tender his resignation when he noticed the model on the side of the server did not match the model he had been told the server was running on. Suddenly, Alex realized the problem - the hosting company had him working on another company’s server.

    It is a small comfort to realize one is not the only idiot.

  • (cs) in reply to foo
    foo:
    boog:
    foo:
    Wow. The article says "after hours", many commenters say "business hours", you just change it to "peak hours" and rely on the definition of "peak".
    Actually, I said "peak hours" in my first answer to you.
    Oh, sorry, my fault. You did actually heed my advice and switch terms early and let some time pass so I actually missed it. Congratulations, you're a senior politician.
    Apparently I'm a politician because you don't pay attention. Wow, you're a freaking genius.
    foo:
    boog:
    foo:
    boog:
    foo:
    OMG, I offended the big boog, and now (s)he's shaming me.
    Who said anything about being offended? (Now who's speculating?)
    I never denied speculating. Though based on your reaction it seems a rather safe guess.
    Who said you denied speculating?
    Who said anyone said I denied speculating?
    Nobody. That's the point.
    foo:
    boog:
    foo:
    OK, so you're not stating facts (since you can't prove them)
    I'm not sure what facts you think I need to prove to you.
    Well you misunderstood me. I didn't say you need to prove things (since I don't think you can). I just pointed out (though somewhat implicitly -- sorry I assumed you could figure this out yourself) that statements can be grouped into 3 categories: 1. proved => true 2. disproved => false 3. neither proved nor disproved => speculation Since your statements obviously aren't 1. and you denied 3., this logially leaves only 2. by your assertion. (I'm still willing to give you 3., but if you don't want ...)
    At which point did I deny 3? Please be specific.

    I did say I was missing the part that was speculation (giving you the opportunity to clarify) - at first it seemed you were saying the idea of most users being on during peak hours was speculation, which is absurd (since that's what peak hours means). But rather than clarify, you threw out hints that only seemed to confirm my initial assumption.

    Since then I realized that the speculation was actually my assumption that office hours coincided with peak hours. Still, I haven't heard you say that (despite giving you ample opportunity) - and why would you? If you did that, I might have said "oh, gotcha, yes you're right", and the discussion would have been over. You'd have missed out on the chance to pretend you were such a smart guy.

  • foo (unregistered) in reply to boog
    boog:
    Apparently I'm a politician because you don't pay attention.
    Because you manipulate with rhetorics instead of arguments. Yes, I didn't pay enough attention to language details because I was focusing on arguments not rhetorics.
    boog:
    foo:
    boog:
    foo:
    boog:
    foo:
    OMG, I offended the big boog, and now (s)he's shaming me.
    Who said anything about being offended? (Now who's speculating?)
    I never denied speculating. Though based on your reaction it seems a rather safe guess.
    Who said you denied speculating?
    Who said anyone said I denied speculating?
    Nobody. That's the point.
    So why did you ask who's speculating? You just wanted me to spell it out (again)? We're both speculating.
    boog:
    foo:
    Well you misunderstood me. I didn't say you need to prove things (since I don't think you can). I just pointed out (though somewhat implicitly -- sorry I assumed you could figure this out yourself) that statements can be grouped into 3 categories: 1. proved => true 2. disproved => false 3. neither proved nor disproved => speculation Since your statements obviously aren't 1. and you denied 3., this logially leaves only 2. by your assertion. (I'm still willing to give you 3., but if you don't want ...)
    At which point did I deny 3? Please be specific. I did say I was missing the part that was speculation (giving you the opportunity to clarify)
    Which I did. I highlighted your "most customers" assertion. I thought it should have been obvious to you at that point that this was just your unverified claim because at that point I was still talking about business hours rather than peak hours and was assuming so were you.
    boog:
    Since then I realized that the speculation was actually my assumption that office hours coincided with peak hours. Still, I haven't heard you say that (despite giving you ample opportunity) - and why would you? If you did that, I might have said "oh, gotcha, yes you're right", and the discussion would have been over.
    As I admitted (above and previously), I had initially missed your change of topic. Now that that's been established, what more should I need to point out? Why didn't you just say, "oh sorry, I answered the wrong question". That would have given you some credibility to claim it was unintentional. Since you missed that opportunity, I assume it was intentional and that's why I call you a politician (because politicians like to play such tricks).
  • mr grammar (unregistered)

    Orders weren't lost. Orders being lost would mean they were placed in the first place and then lost. These weren't. This company missed\lost out on..a days worth of orders.

  • yername (unregistered) in reply to ronpaii
    ronpaii:
    So Alex makes a rooky error. The hosting company shuts down the wrong server causing a 3 hour outage for your company and another. Alex drives to the site, finds the error and fixes it. And you put all this on Alex?
    Nah, he's just degenerated to a PHB.
  • anonymous (unregistered) in reply to boog

    The only troll I see here is you.

  • Stephen Green (unregistered)

    Uh yea, senior management looks to save money. Period. Working for a living stinks. No matter how good YOU are..

  • (cs) in reply to Hans
    Hans:
    The command sequence "/etc/init.d/network stop ; /etc/init.d/network start" DOES work.

    The system will not immediately kill the connection just because the network device goes down, it would only time-out after a while. Also the shell already got the complete command sequence and will execute it till completion.

    Just try it, it works. So this story does not make sense from the beginning and is probably just a untrue tale. Sad.

    Try that again with a network change that will make the current connection invalid. Depending on your operating system, at best, your connection dies, but the start command completes. But the OS on which I ran that command-line remotely without screen (I only ever did it once), it terminated the ssh connection and HUPped everything running on the TTY it opened. That left the box with a running network. YAY!

    '/etc/init.d/network stop' killed all of the server parent processes - sshd, sendmail, apache, etc.

    '/etc/init.d/network start' was killed before it started any server parent processes.

    So the box was on the network, but still inaccessible. Not a distinction that most people would make, as it's effectively offline. But it's crucial to understanding exactly why the process failed.

    Running '/etc/init.d/network restart' instead wouldn't have helped - it still would've killed everything on the TTY. Similarly, '(/etc/init.d/network stop; /etc/init.d/network start)&' would not have helped.

    Running '/etc/init.d/network stop; nohup /etc/init.d/network start' would have. Running the command-line in a screen session would have also helped.

    Note when I made my blunder, I was just down the hallway from the computer. It was back in service in about five minutes. (Long hallway, but I ran. Big, mostly empty building, so I could get away with that.) We did have remote terminals - which worked on all of our equipment except that one. Unfortunately, someone thought it'd be a good idea to give the box an array that required the serial port for a separate command interface, and it only had one serial port, and the terminal switch only did serial port consoles. Sigh.

Leave a comment on “Remotely Incompetent”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article