• (cs)

    I suggest they put up a note in the room with this text:

    ACHTUNG! ALLES TURISTEN UND NONTEKNISCHEN LOOKENPEEPERS! DAS KOMPUTERMASCHINE IST NICHT FÜR DER GEFINGERPOKEN UND MITTENGRABEN! ODERWISE IST EASY TO SCHNAPPEN DER SPRINGENWERK, BLOWENFUSEN UND POPPENCORKEN MIT SPITZENSPARKSEN. IST NICHT FÜR GEWERKEN BEI DUMMKOPFEN. DER RUBBERNECKEN SIGHTSEEREN KEEPEN DAS COTTONPICKEN HÄNDER IN DAS POCKETS MUSS. ZO RELAXEN UND WATSCHEN DER BLINKENLICHTEN.

    PS: What The Fr1st!

  • Brian (unregistered)

    I would think a better use of paper would be to put an ad in the jobs section of the local newspaper for a competent IT data center tech. How long did it take to track this problem down????

  • Nagesh (unregistered)

    You know you've been reading TDWTF too long when you can read the first two paragraphs of an article, and the only suspense left is whether it was something plugged into the wrong socket, or somebody explicitly unplugging it in favor of a kettle/phone charger/vacuum cleaner.

  • (cs) in reply to ochrist
    ochrist:
    I suggest they put up a note in the room with this text:

    ACHTUNG! ALLES TURISTEN UND NONTEKNISCHEN LOOKENPEEPERS! DAS KOMPUTERMASCHINE IST NICHT FÜR DER GEFINGERPOKEN UND MITTENGRABEN! ODERWISE IST EASY TO SCHNAPPEN DER SPRINGENWERK, BLOWENFUSEN UND POPPENCORKEN MIT SPITZENSPARKSEN. IST NICHT FÜR GEWERKEN BEI DUMMKOPFEN. DER RUBBERNECKEN SIGHTSEEREN KEEPEN DAS COTTONPICKEN HÄNDER IN DAS POCKETS MUSS. ZO RELAXEN UND WATSCHEN DER BLINKENLICHTEN.

    PS: What The Fr1st!

    You made my Monday!

  • Your Name (unregistered) in reply to Brian
    Brian:
    I would think a better use of paper would be to put an ad in the jobs section of the local newspaper for a competent IT data center tech. How long did it take to track this problem down????

    The problem is always something stupidly simple, because you try to solve the complex ones first.

  • CodeCaster (unregistered) in reply to Nagesh
    Nagesh:
    You know you've been reading TDWTF too long when you can read the first two paragraphs of an article, and the only suspense left is whether it was something plugged into the wrong socket, or somebody explicitly unplugging it in favor of a kettle/phone charger/vacuum cleaner.
    Here, here.
  • Studley (unregistered)
    True to her word, she never went anywhere near the servers, and at 12PM, she grabbed her purse, jammed the sudoku puzzles in there, and left.
    That must have been a very tough sudoku puzzle, to keep her in the office until noon?
  • Sten (unregistered)

    TRWTF is the electrician who connected an outlet to the lights. AFAIK it is illegal in the EU for safety reasons

  • Roy (unregistered)

    I was expecting a vacuumcleaner in a wrong socket... but close enough ;)

  • Matteo (unregistered)

    Actually I already knew this anecdote (and many others) from "How Not to Program in C++", by Steve Oualline. This is the long version, but it appears it's not a new story.

  • Some Jerk (unregistered) in reply to ochrist
    ochrist:
    I suggest they put up a note in the room with this text:

    ACHTUNG! ALLES TURISTEN UND NONTEKNISCHEN LOOKENPEEPERS! DAS KOMPUTERMASCHINE IST NICHT FÜR DER GEFINGERPOKEN UND MITTENGRABEN! ODERWISE IST EASY TO SCHNAPPEN DER SPRINGENWERK, BLOWENFUSEN UND POPPENCORKEN MIT SPITZENSPARKSEN. IST NICHT FÜR GEWERKEN BEI DUMMKOPFEN. DER RUBBERNECKEN SIGHTSEEREN KEEPEN DAS COTTONPICKEN HÄNDER IN DAS POCKETS MUSS. ZO RELAXEN UND WATSCHEN DER BLINKENLICHTEN.

    PS: What The Fr1st!

    FUT DA WHAKIN... MUC TU READINBLEADIN

  • Carl (unregistered)
    It never finished, and it hadn't for over a year. Each day, somebody spent an hour or two cleaning up after its inevitable, messy failures.
    And in all that time, nobody noticed the following pattern?

    ... success success success fail fail fail ...

    as opposed to

    ... success fail fail success fail success success ...

    In the latter case you have something intermittent; in the story the failures should have always happened in one batch at the tail end of the job. Seems that would be a fairly large clue.

    Anyway, TRWTF is using FTP instead of rsync.

  • oldami (unregistered)

    Agree this should not have taken long to figure out. Even the simplest remote server monitor program would have detected the server dropping off-line when the the switch powered off. Once you know that, there are only a few things to verify are working.

  • Some Jerk (unregistered) in reply to Carl

    Now... they might wish to move the building security system from that socket.

    CAPTCHA: ingenium - FER TRU INGENUIM, WUZ REEDIN BLEEDIN

  • Come on. Seriously? (unregistered)

    Nobody was able to determine network connectivity issues were to blame? Seriously? That would have been one of the first things I'd have checked. If nothing else,

    ping /t example.com > pings.txt

    I call the whole thing BS.

  • (cs) in reply to ochrist
    ochrist:
    I suggest they put up a note in the room with this text:

    ACHTUNG! ALLES TURISTEN UND NONTEKNISCHEN LOOKENPEEPERS! DAS KOMPUTERMASCHINE IST NICHT FÜR DER GEFINGERPOKEN UND MITTENGRABEN! ODERWISE IST EASY TO SCHNAPPEN DER SPRINGENWERK, BLOWENFUSEN UND POPPENCORKEN MIT SPITZENSPARKSEN. IST NICHT FÜR GEWERKEN BEI DUMMKOPFEN. DER RUBBERNECKEN SIGHTSEEREN KEEPEN DAS COTTONPICKEN HÄNDER IN DAS POCKETS MUSS. ZO RELAXEN UND WATSCHEN DER BLINKENLICHTEN.

    PS: What The Fr1st!

    Best frist post since so long... I didn't even know one could write a first post without being batshit stupid. Congrats !

  • StMarc (unregistered) in reply to Sten

    It's very common in the US (although in newer construction the trend seems to be away from it, thank the evil gods) for a room to have a switched outlet instead of an overhead light. (You plug a lamp into the outlet and that's how you turn the room light on and off with the switch.) I hate this and have rewired every house I've owned to make all the outlets constant-power and added overhead lights, but many people don't seem to have an issue with it.

    The OP actually doesn't say that the room had any overhead lights at all. It's entirely possible the switch controlled multiple outlets. It seems likely that somebody tied an overhead light circuit into the switched outlet circuit, or vice versa, but you never know. And I can think of reasons why you might want to do it. Even more likely - and not violating your rule at all so far as I can tell - is the possibility that there was a bank of switches, one or more of which controlled overhead lights and one of which controlled the switched outlet. People have a tendency to just flip all the switches on a bank when entering or leaving a room. (And don't ask me about the place I used to work where they used circuit breakers as light switches.)

    What, exactly, is the safety issue to which you refer? I can think of a few examples, but none of them really seem so terrifying as to call for a code requirement. More like a "don't be a cheap moron" requirement.

  • foxyshadis (unregistered) in reply to Come on. Seriously?
    Come on. Seriously?:
    Nobody was able to determine network connectivity issues were to blame? Seriously? That would have been one of the first things I'd have checked. If nothing else,

    ping /t example.com > pings.txt

    I call the whole thing BS.

    What do you expect, when you have a boss who orders you not to solve it just because some other schmuck couldn't do basic troubleshooting? Also, it appears that the sysadmins might have never been informed, if it was only programmers who were looking into the problem all along.

  • StMarc (unregistered) in reply to StMarc

    Sorry, that was in response to:

    Sten:
    TRWTF is the electrician who connected an outlet to the lights. AFAIK it is illegal in the EU for safety reasons

  • Some Jerk (unregistered) in reply to StMarc

    unsafe if you have your fire detection plugged into that spot.

  • (cs) in reply to Come on. Seriously?

    If someone had checked network connectivity, this story wouldn't be running here, now would it?

  • ¯\(°_o)/¯ I DUNNO LOL (unregistered) in reply to StMarc
    StMarc:
    It's very common in the US (although in newer construction the trend seems to be away from it, thank the evil gods) for a room to have a switched outlet instead of an overhead light.
    In a residential building, sure. In a commercial building, no way.
  • Le Forgeron (unregistered) in reply to Remy Porter
    Remy Porter:
    If someone had checked network connectivity, this story wouldn't be running here, now would it?

    But network connectivity was there, in normal daytime. Even at the start of the night, it was still there.

    And as you came in the room early in the morning, it was already back as you turn the light on...

    Yet the ftp program of the night failed.

  • Some Jerk (unregistered) in reply to ¯\(°_o)/¯ I DUNNO LOL

    A government office... trying to eek a few cents off of the electric bill

  • foo (unregistered) in reply to Some Jerk
    Some Jerk:
    unsafe if you have your fire detection plugged into that spot.
    This, or something like: "Hmm, this socket is dead, I'll have to open it and check the wires. But let me just turn on the lights so I can see better ..."
  • JC (unregistered)

    I here you loud and clear

    • weeps *

    "Herwig grimaced, if only because a roving HR drone might here that and draw the wrong conclusion."

  • foo (unregistered) in reply to Carl
    Carl:
    It never finished, and it hadn't for over a year. Each day, somebody spent an hour or two cleaning up after its inevitable, messy failures.
    And in all that time, nobody noticed the following pattern?

    ... success success success fail fail fail ...

    as opposed to

    ... success fail fail success fail success success ...

    In the latter case you have something intermittent; in the story the failures should have always happened in one batch at the tail end of the job. Seems that would be a fairly large clue.

    AIUI, there's only one job per night.
    Anyway, TRWTF is using FTP instead of rsync.
    Ack. Though I still wonder what of kind of terrible mess FTP could possibly make. At worst, there should be one partly transmitted file and the rest of the files missing. Retransmit them and done.

  • Some Jerk (unregistered) in reply to foo

    Don't worry... I charged my backup pacemaker over the weekend.... now I'll drive us to <insert destination here>

  • (cs)

    I still think it's a WTF that desktop computers and servers are not like laptops with their own internal power source with the mains acting as just a recharger.

    Even if they were not able to hold power for a period of time (due to consuming far more energy than laptops) they could maintain power for a short period in order to be able to "hibernate" by persisting the current state and resume as soon as the main power source came back.

    It's also a WTF that they are paying someone to be onsite until midnight just doing sukodu puzzles.

    Incidentally if Greta wants to make some career progress she would better spend her time reading some technical books or browsing technical websites and then find herself an alternative job.

  • Skeeve (unregistered) in reply to foo
    foo:
    This, or something like: "Hmm, this socket is dead, I'll have to open it and check the wires. But let me just turn on the lights so I can see better ..."

    Anyone who opens up a socket to handle the wires without turning the circuit off at the breaker is a moron who deserves to have their electrician's licence revoked, for the safety of everyone around them.

    Seriously. Switched or not, you turn the power off before you screw around with the wiring, no matter what. Ignoring that is a pretty good way to end up hurt.

  • Franky (unregistered)

    actually, I have the same configuration for my TV ... which is awesome, I just walk into the room, carelessly flip the light-switch on doing so, and the tv springs to live ... everybody is "wtf? how did you do that" the first time they see it :D

    and the best is, the thing doesn't draw any excess power when it is switched off (which is most of the day)

  • (cs) in reply to Carl
    Carl:
    Anyway, TRWTF is using FTP instead of rsync.

    This.

  • (cs) in reply to Cbuttius
    Cbuttius:
    I still think it's a WTF that desktop computers and servers are not like laptops with their own internal power source with the mains acting as just a recharger.
    It's called a UPS, and it's a feature of systems managed by someone who's actually competent.
    Cbuttius:
    Even if they were not able to hold power for a period of time (due to consuming far more energy than laptops) they could maintain power for a short period in order to be able to "hibernate" by persisting the current state and resume as soon as the main power source came back.
    Assuming that they've been wired up right, they only need to hold power until the onsite generator comes up and stabilizes. Of course, experience of operations at a national datacenter tells me that the trick is to remember to put the network routers and AC on the generator too. And to remember to check that the upstream ISP has done the same thing…
  • (cs) in reply to oldami
    oldami:
    Agree this should not have taken long to figure out. Even the simplest remote server monitor program would have detected the server dropping off-line when the the switch powered off. Once you know that, there are only a few things to verify are working.
    A complete classic would be if there were two switches, one for the internal network and another for the external network. Then if it was just the external switch that lost power, the internal network and the monitoring system could stay up and functioning but the file transfer would still fail. (Multiple network interfaces per server isn't exactly unheard of.)
  • badman (unregistered) in reply to Cbuttius

    I used to work at a chemical treatment facility where I was paid to sit and read all night. I just had to listen for buzzers and watch for red lights. Also every hour or so I would walk thru the facility and look for anything that "didn't look right". Of course I was trained, and knew how to handle thing should something go amiss, but it hardly ever did. an in the many years I had that job I only had to pick up "the red phone" and make the call to the government once. You know the one, "this is operator xxxx at site yyyy and I would like to log an incident"......

  • (cs) in reply to dkf
    dkf:
    Cbuttius:
    I still think it's a WTF that desktop computers and servers are not like laptops with their own internal power source with the mains acting as just a recharger.
    It's called a UPS, and it's a feature of systems managed by someone who's actually competent.
    Cbuttius:
    Even if they were not able to hold power for a period of time (due to consuming far more energy than laptops) they could maintain power for a short period in order to be able to "hibernate" by persisting the current state and resume as soon as the main power source came back.
    Assuming that they've been wired up right, they only need to hold power until the onsite generator comes up and stabilizes. Of course, experience of operations at a national datacenter tells me that the trick is to remember to put the network routers and AC on the generator too. And to remember to check that the upstream ISP has done the same thing…

    I am not competent in that field, I am very much a software engineer and not a hardware expert. However I am competent enough to be able to provide a minimum spec of what I would like my computer to behave if the power cable is pulled from it.

  • Ben Jammin (unregistered)

    It's that he's concerned with HR about:

    "That's what 'night-shift' means," Greta replied with a smirk. "You were here for the whole thing last night."

    But not concerned about hr/police about:

    With the lights off on his side, Greta couldn't see him, but he could watch her through the glass walls.
  • Ben Jammin (unregistered) in reply to Ben Jammin

    funny*

    My mind types faster than my fingers

  • (cs) in reply to Cbuttius

    "I still think it's a WTF that desktop computers and servers are not like laptops with their own internal power source with the mains acting as just a recharger."

    It's a solved problem for servers, as any competently designed server room will already have everything critical getting its power through a UPS. In the case in this story, it's possible that the server room proper has a UPS, but the overflow in the Copy Center does not.

    It's kind of a solved problem for desktops, too, as home UPS units capable of gently shutting down a home computer and its peripherals in the event of an outage can be bought for under $100.

    What I don't understand is why more server rooms don't let the UPSes deliver DC voltage direct to the server motherboards at all times.

  • Infinite Time and Space (unregistered) in reply to ochrist
    ochrist:
    I suggest they put up a note in the room with this text:

    ACHTUNG! ALLES TURISTEN UND NONTEKNISCHEN LOOKENPEEPERS! DAS KOMPUTERMASCHINE IST NICHT FÜR DER GEFINGERPOKEN UND MITTENGRABEN! ODERWISE IST EASY TO SCHNAPPEN DER SPRINGENWERK, BLOWENFUSEN UND POPPENCORKEN MIT SPITZENSPARKSEN. IST NICHT FÜR GEWERKEN BEI DUMMKOPFEN. DER RUBBERNECKEN SIGHTSEEREN KEEPEN DAS COTTONPICKEN HÄNDER IN DAS POCKETS MUSS. ZO RELAXEN UND WATSCHEN DER BLINKENLICHTEN.

    PS: What The Fr1st!

    Two years of college German and that is about as well as I can speak it.

  • (cs) in reply to Rootbeer
    Rootbeer:
    "I still think it's a WTF that desktop computers and servers are not like laptops with their own internal power source with the mains acting as just a recharger."

    It's a solved problem for servers, as any competently designed server room...

    ...will not end up on this site.

  • ufmace (unregistered)

    First response to reading: the standard chuckle

    Second response: Wait a second, how did they not figure that out sooner? What kind of troubleshooting plan never ruled out a loss of connectivity to the server? What kind of logging system never noted the type of error? While we're at it, what kind of manager tells their employees to not bother trying to fix a simple problem like that? Groan.

  • Mikerad (unregistered) in reply to Ben Jammin
    Ben Jammin:
    It's that he's concerned with HR about:
    "That's what 'night-shift' means," Greta replied with a smirk. "You were here for the whole thing last night."

    But not concerned about hr/police about:

    With the lights off on his side, Greta couldn't see him, but he could watch her through the glass walls.

    Everyone knows HR doesn't exist after 5pm...

  • EmptyJay (unregistered) in reply to StMarc

    Amen.

    I can't stand not having built-in overhead lights. We installed them in both bedrooms before we moved into our current place, and I added recessed lights to the living room after about 2 years of suffering without.

    I still find the switched outlets to be useful, but I'm slowly converting them to constant power.

  • (cs) in reply to Some Jerk
    Some Jerk:
    unsafe if you have your fire detection plugged into that spot.
    Pretty much handled by the NEC code that says you must have a dedicated circuit for fire detection and alarm equipment, and it can't have a switch on it. Try again.
  • (cs) in reply to foo
    foo:
    Some Jerk:
    unsafe if you have your fire detection plugged into that spot.
    This, or something like: "Hmm, this socket is dead, I'll have to open it and check the wires. But let me just turn on the lights so I can see better ..."
    Like when handling a firearm, always assume the circuit is live unless you've flipped the breaker and tested for current at the receptacle. Do they have untrained morons playing with electricity in the EU?
  • foo (unregistered) in reply to Cbuttius
    Cbuttius:
    dkf:
    Cbuttius:
    I still think it's a WTF that desktop computers and servers are not like laptops with their own internal power source with the mains acting as just a recharger.
    It's called a UPS, and it's a feature of systems managed by someone who's actually competent.
    Cbuttius:
    Even if they were not able to hold power for a period of time (due to consuming far more energy than laptops) they could maintain power for a short period in order to be able to "hibernate" by persisting the current state and resume as soon as the main power source came back.
    Assuming that they've been wired up right, they only need to hold power until the onsite generator comes up and stabilizes. Of course, experience of operations at a national datacenter tells me that the trick is to remember to put the network routers and AC on the generator too. And to remember to check that the upstream ISP has done the same thing…

    I am not competent in that field, I am very much a software engineer and not a hardware expert. However I am competent enough to be able to provide a minimum spec of what I would like my computer to behave if the power cable is pulled from it.

    As others said, you can have it. It's just a matter of cost. Many people (including myself :) don't spend the money, since power outages are very rare where I live, and journaled file systems avert most problems if they do occur.

    However, note that this story was about the network switch losing power. Shutting it down gently wouldn't help a bit.

  • (cs) in reply to operagost

    I've done some wiring with the breaker on. It's the kind of mistake that you only make two or three times before you finally learn your lesson. Just call me "Sparky".

    //Kidding- the only times I've gotten shocked from an outlet have been for far dumber reasons than not turning off the breaker.

  • foo (unregistered) in reply to operagost
    operagost:
    foo:
    Some Jerk:
    unsafe if you have your fire detection plugged into that spot.
    This, or something like: "Hmm, this socket is dead, I'll have to open it and check the wires. But let me just turn on the lights so I can see better ..."
    Like when handling a firearm, always assume the circuit is live unless you've flipped the breaker and tested for current at the receptacle. Do they have untrained morons playing with electricity in the EU?
    Sure. Don't they in the US (or anywhere else)? I mean people working on their own home's installations; sure, they're often stupid, and saftely regulations just add one layer of protection.
  • Some Jerk (unregistered) in reply to operagost
    operagost:
    Some Jerk:
    unsafe if you have your fire detection plugged into that spot.
    Pretty much handled by the NEC code that says you must have a dedicated circuit for fire detection and alarm equipment, and it can't have a switch on it. Try again.

    that was intended for humor dude. Sort of like charging the pacemaker. TRY AGAIN :p

Leave a comment on “The Long Goodbye”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article