Comment On The Long Goodbye

Herwig smiled at Greta as he entered the glass-walled copy-center. "Excuse me, but do you mind if I ask you a few questions?" [expand full text]
« PrevPage 1 | Page 2 | Page 3Next »

Re: The Long Goodbye

2012-08-06 08:25 • by ochrist
I suggest they put up a note in the room with this text:

ACHTUNG!
ALLES TURISTEN UND NONTEKNISCHEN LOOKENPEEPERS!
DAS KOMPUTERMASCHINE IST NICHT FÜR DER GEFINGERPOKEN UND MITTENGRABEN! ODERWISE IST EASY TO SCHNAPPEN DER SPRINGENWERK, BLOWENFUSEN UND POPPENCORKEN MIT SPITZENSPARKSEN.
IST NICHT FÜR GEWERKEN BEI DUMMKOPFEN. DER RUBBERNECKEN SIGHTSEEREN KEEPEN DAS COTTONPICKEN HÄNDER IN DAS POCKETS MUSS.
ZO RELAXEN UND WATSCHEN DER BLINKENLICHTEN.

PS: What The Fr1st!

Re: The Long Goodbye

2012-08-06 08:28 • by Brian (unregistered)
I would think a better use of paper would be to put an ad in the jobs section of the local newspaper for a competent IT data center tech. How long did it take to track this problem down????

Re: The Long Goodbye

2012-08-06 08:34 • by Nagesh (unregistered)
You know you've been reading TDWTF too long when you can read the first two paragraphs of an article, and the only suspense left is whether it was something plugged into the wrong socket, or somebody explicitly unplugging it in favor of a kettle/phone charger/vacuum cleaner.

Re: The Long Goodbye

2012-08-06 08:35 • by snoofle
386429 in reply to 386424
ochrist:
I suggest they put up a note in the room with this text:

ACHTUNG!
ALLES TURISTEN UND NONTEKNISCHEN LOOKENPEEPERS!
DAS KOMPUTERMASCHINE IST NICHT FÜR DER GEFINGERPOKEN UND MITTENGRABEN! ODERWISE IST EASY TO SCHNAPPEN DER SPRINGENWERK, BLOWENFUSEN UND POPPENCORKEN MIT SPITZENSPARKSEN.
IST NICHT FÜR GEWERKEN BEI DUMMKOPFEN. DER RUBBERNECKEN SIGHTSEEREN KEEPEN DAS COTTONPICKEN HÄNDER IN DAS POCKETS MUSS.
ZO RELAXEN UND WATSCHEN DER BLINKENLICHTEN.

PS: What The Fr1st!
You made my Monday!

Re: The Long Goodbye

2012-08-06 08:37 • by Your Name (unregistered)
386430 in reply to 386425
Brian:
I would think a better use of paper would be to put an ad in the jobs section of the local newspaper for a competent IT data center tech. How long did it take to track this problem down????


The problem is always something stupidly simple, because you try to solve the complex ones first.

Re: The Long Goodbye

2012-08-06 08:39 • by CodeCaster (unregistered)
386431 in reply to 386428
Nagesh:
You know you've been reading TDWTF too long when you can read the first two paragraphs of an article, and the only suspense left is whether it was something plugged into the wrong socket, or somebody explicitly unplugging it in favor of a kettle/phone charger/vacuum cleaner.

Here, here.

Re: The Long Goodbye

2012-08-06 08:42 • by Studley (unregistered)
True to her word, she never went anywhere near the servers, and at 12PM, she grabbed her purse, jammed the sudoku puzzles in there, and left.
That must have been a very tough sudoku puzzle, to keep her in the office until noon?

Re: The Long Goodbye

2012-08-06 08:43 • by Sten (unregistered)
TRWTF is the electrician who connected an outlet to the lights. AFAIK it is illegal in the EU for safety reasons

Re: The Long Goodbye

2012-08-06 08:45 • by Roy (unregistered)
I was expecting a vacuumcleaner in a wrong socket... but close enough ;)

Re: The Long Goodbye

2012-08-06 08:51 • by Matteo (unregistered)
Actually I already knew this anecdote (and many others) from "How Not to Program in C++", by Steve Oualline. This is the long version, but it appears it's not a new story.

Re: The Long Goodbye

2012-08-06 08:53 • by Some Jerk (unregistered)
386436 in reply to 386424
ochrist:
I suggest they put up a note in the room with this text:

ACHTUNG!
ALLES TURISTEN UND NONTEKNISCHEN LOOKENPEEPERS!
DAS KOMPUTERMASCHINE IST NICHT FÜR DER GEFINGERPOKEN UND MITTENGRABEN! ODERWISE IST EASY TO SCHNAPPEN DER SPRINGENWERK, BLOWENFUSEN UND POPPENCORKEN MIT SPITZENSPARKSEN.
IST NICHT FÜR GEWERKEN BEI DUMMKOPFEN. DER RUBBERNECKEN SIGHTSEEREN KEEPEN DAS COTTONPICKEN HÄNDER IN DAS POCKETS MUSS.
ZO RELAXEN UND WATSCHEN DER BLINKENLICHTEN.

PS: What The Fr1st!


FUT DA WHAKIN... MUC TU READINBLEADIN

Re: The Long Goodbye

2012-08-06 08:58 • by Carl (unregistered)
It never finished, and it hadn't for over a year. Each day, somebody spent an hour or two cleaning up after its inevitable, messy failures.
And in all that time, nobody noticed the following pattern?

...
success
success
success
fail
fail
fail
...

as opposed to

...
success
fail
fail
success
fail
success
success
...

In the latter case you have something intermittent; in the story the failures should have always happened in one batch at the tail end of the job. Seems that would be a fairly large clue.

Anyway, TRWTF is using FTP instead of rsync.

Re: The Long Goodbye

2012-08-06 08:58 • by oldami (unregistered)
Agree this should not have taken long to figure out. Even the simplest remote server monitor program would have detected the server dropping off-line when the the switch powered off. Once you know that, there are only a few things to verify are working.

Re: The Long Goodbye

2012-08-06 09:02 • by Some Jerk (unregistered)
386439 in reply to 386437
Now... they might wish to move the building security system from that socket.

CAPTCHA: ingenium - FER TRU INGENUIM, WUZ REEDIN BLEEDIN

Re: The Long Goodbye

2012-08-06 09:03 • by Come on. Seriously? (unregistered)
Nobody was able to determine network connectivity issues were to blame? Seriously? That would have been one of the first things I'd have checked. If nothing else,

ping /t example.com > pings.txt

I call the whole thing BS.

Re: The Long Goodbye

2012-08-06 09:03 • by toshir0
386441 in reply to 386424
ochrist:
I suggest they put up a note in the room with this text:

ACHTUNG!
ALLES TURISTEN UND NONTEKNISCHEN LOOKENPEEPERS!
DAS KOMPUTERMASCHINE IST NICHT FÜR DER GEFINGERPOKEN UND MITTENGRABEN! ODERWISE IST EASY TO SCHNAPPEN DER SPRINGENWERK, BLOWENFUSEN UND POPPENCORKEN MIT SPITZENSPARKSEN.
IST NICHT FÜR GEWERKEN BEI DUMMKOPFEN. DER RUBBERNECKEN SIGHTSEEREN KEEPEN DAS COTTONPICKEN HÄNDER IN DAS POCKETS MUSS.
ZO RELAXEN UND WATSCHEN DER BLINKENLICHTEN.

PS: What The Fr1st!
Best *frist* post since so long... I didn't even know one could write a first post without being batshit stupid. Congrats !

Re: The Long Goodbye

2012-08-06 09:14 • by StMarc (unregistered)
386442 in reply to 386433
It's very common in the US (although in newer construction the trend seems to be away from it, thank the evil gods) for a room to have a switched outlet instead of an overhead light. (You plug a lamp into the outlet and that's how you turn the room light on and off with the switch.) I hate this and have rewired every house I've owned to make all the outlets constant-power and added overhead lights, but many people don't seem to have an issue with it.

The OP actually doesn't *say* that the room had any overhead lights at all. It's entirely possible the switch controlled multiple outlets. It seems likely that somebody tied an overhead light circuit into the switched outlet circuit, or vice versa, but you never know. And I can think of reasons why you might want to do it. Even more likely - and not violating your rule at all so far as I can tell - is the possibility that there was a *bank* of switches, one or more of which controlled overhead lights and one of which controlled the switched outlet. People have a tendency to just flip all the switches on a bank when entering or leaving a room. (And don't ask me about the place I used to work where they used circuit breakers as light switches.)

What, exactly, is the safety issue to which you refer? I can think of a few examples, but none of them really seem so terrifying as to call for a code requirement. More like a "don't be a cheap moron" requirement.

Re: The Long Goodbye

2012-08-06 09:14 • by foxyshadis (unregistered)
386443 in reply to 386440
Come on. Seriously?:
Nobody was able to determine network connectivity issues were to blame? Seriously? That would have been one of the first things I'd have checked. If nothing else,

ping /t example.com > pings.txt

I call the whole thing BS.

What do you expect, when you have a boss who orders you *not* to solve it just because some other schmuck couldn't do basic troubleshooting? Also, it appears that the sysadmins might have never been informed, if it was only programmers who were looking into the problem all along.

Re: The Long Goodbye

2012-08-06 09:16 • by StMarc (unregistered)
386444 in reply to 386442
Sorry, that was in response to:
Sten:
TRWTF is the electrician who connected an outlet to the lights. AFAIK it is illegal in the EU for safety reasons

Re: The Long Goodbye

2012-08-06 09:19 • by Some Jerk (unregistered)
386445 in reply to 386442
unsafe if you have your fire detection plugged into that spot.

Re: The Long Goodbye

2012-08-06 09:25 • by Remy Porter
386446 in reply to 386440
If someone had checked network connectivity, this story wouldn't be running here, now would it?

Re: The Long Goodbye

2012-08-06 09:26 • by ¯\(°_o)/¯ I DUNNO LOL (unregistered)
386447 in reply to 386442
StMarc:
It's very common in the US (although in newer construction the trend seems to be away from it, thank the evil gods) for a room to have a switched outlet instead of an overhead light.
In a residential building, sure. In a commercial building, no way.

Re: The Long Goodbye

2012-08-06 09:33 • by Le Forgeron (unregistered)
386448 in reply to 386446
Remy Porter:
If someone had checked network connectivity, this story wouldn't be running here, now would it?


But network connectivity was there, in normal daytime.
Even at the start of the night, it was still there.

And as you came in the room early in the morning, it was already back as you turn the light on...

Yet the ftp program of the night failed.

Re: The Long Goodbye

2012-08-06 09:34 • by Some Jerk (unregistered)
386449 in reply to 386447
A government office... trying to eek a few cents off of the electric bill

Re: The Long Goodbye

2012-08-06 09:40 • by foo (unregistered)
386451 in reply to 386445
Some Jerk:
unsafe if you have your fire detection plugged into that spot.
This, or something like: "Hmm, this socket is dead, I'll have to open it and check the wires. But let me just turn on the lights so I can see better ..."

Re: The Long Goodbye

2012-08-06 09:40 • by JC (unregistered)
I here you loud and clear

* weeps *

"Herwig grimaced, if only because a roving HR drone might here that and draw the wrong conclusion."

Re: The Long Goodbye

2012-08-06 09:41 • by foo (unregistered)
386453 in reply to 386437
Carl:
It never finished, and it hadn't for over a year. Each day, somebody spent an hour or two cleaning up after its inevitable, messy failures.
And in all that time, nobody noticed the following pattern?

...
success
success
success
fail
fail
fail
...

as opposed to

...
success
fail
fail
success
fail
success
success
...

In the latter case you have something intermittent; in the story the failures should have always happened in one batch at the tail end of the job. Seems that would be a fairly large clue.
AIUI, there's only one job per night.
Anyway, TRWTF is using FTP instead of rsync.
Ack. Though I still wonder what of kind of terrible mess FTP could possibly make. At worst, there should be one partly transmitted file and the rest of the files missing. Retransmit them and done.

Re: The Long Goodbye

2012-08-06 09:52 • by Some Jerk (unregistered)
386455 in reply to 386451
Don't worry... I charged my backup pacemaker over the weekend.... now I'll drive us to <insert destination here>

Re: The Long Goodbye

2012-08-06 10:00 • by Cbuttius
I still think it's a WTF that desktop computers and servers are not like laptops with their own internal power source with the mains acting as just a recharger.

Even if they were not able to hold power for a period of time (due to consuming far more energy than laptops) they could maintain power for a short period in order to be able to "hibernate" by persisting the current state and resume as soon as the main power source came back.

It's also a WTF that they are paying someone to be onsite until midnight just doing sukodu puzzles.

Incidentally if Greta wants to make some career progress she would better spend her time reading some technical books or browsing technical websites and then find herself an alternative job.

Re: The Long Goodbye

2012-08-06 10:08 • by Skeeve (unregistered)
386458 in reply to 386451
foo:
This, or something like: "Hmm, this socket is dead, I'll have to open it and check the wires. But let me just turn on the lights so I can see better ..."


Anyone who opens up a socket to handle the wires without turning the circuit off at the breaker is a moron who deserves to have their electrician's licence revoked, for the safety of everyone around them.

Seriously. Switched or not, you turn the power off before you screw around with the wiring, no matter what. Ignoring that is a pretty good way to end up hurt.

Re: The Long Goodbye

2012-08-06 10:09 • by Franky (unregistered)
actually, I have the same configuration for my TV ... which is awesome, I just walk into the room, carelessly flip the light-switch on doing so, and the tv springs to live ... everybody is "wtf? how did you do that" the first time they see it :D

and the best is, the thing doesn't draw any excess power when it is switched off (which is most of the day)

Re: The Long Goodbye

2012-08-06 10:10 • by java.lang.Chris;
386460 in reply to 386437
Carl:
Anyway, TRWTF is using FTP instead of rsync.


This.

Re: The Long Goodbye

2012-08-06 10:12 • by dkf
386461 in reply to 386457
Cbuttius:
I still think it's a WTF that desktop computers and servers are not like laptops with their own internal power source with the mains acting as just a recharger.
It's called a UPS, and it's a feature of systems managed by someone who's actually competent.
Cbuttius:
Even if they were not able to hold power for a period of time (due to consuming far more energy than laptops) they could maintain power for a short period in order to be able to "hibernate" by persisting the current state and resume as soon as the main power source came back.
Assuming that they've been wired up right, they only need to hold power until the onsite generator comes up and stabilizes. Of course, experience of operations at a national datacenter tells me that the trick is to remember to put the network routers and AC on the generator too. And to remember to check that the upstream ISP has done the same thing…

Re: The Long Goodbye

2012-08-06 10:15 • by dkf
386462 in reply to 386438
oldami:
Agree this should not have taken long to figure out. Even the simplest remote server monitor program would have detected the server dropping off-line when the the switch powered off. Once you know that, there are only a few things to verify are working.
A complete classic would be if there were two switches, one for the internal network and another for the external network. Then if it was just the external switch that lost power, the internal network and the monitoring system could stay up and functioning but the file transfer would still fail. (Multiple network interfaces per server isn't exactly unheard of.)

Re: The Long Goodbye

2012-08-06 10:20 • by badman (unregistered)
386463 in reply to 386457
I used to work at a chemical treatment facility where I was paid to sit and read all night. I just had to listen for buzzers and watch for red lights. Also every hour or so I would walk thru the facility and look for anything that "didn't look right". Of course I was trained, and knew how to handle thing should something go amiss, but it hardly ever did. an in the many years I had that job I only had to pick up "the red phone" and make the call to the government once. You know the one, "this is operator xxxx at site yyyy and I would like to log an incident"......

Re: The Long Goodbye

2012-08-06 10:35 • by Cbuttius
386464 in reply to 386461
dkf:
Cbuttius:
I still think it's a WTF that desktop computers and servers are not like laptops with their own internal power source with the mains acting as just a recharger.
It's called a UPS, and it's a feature of systems managed by someone who's actually competent.
Cbuttius:
Even if they were not able to hold power for a period of time (due to consuming far more energy than laptops) they could maintain power for a short period in order to be able to "hibernate" by persisting the current state and resume as soon as the main power source came back.
Assuming that they've been wired up right, they only need to hold power until the onsite generator comes up and stabilizes. Of course, experience of operations at a national datacenter tells me that the trick is to remember to put the network routers and AC on the generator too. And to remember to check that the upstream ISP has done the same thing…


I am not competent in that field, I am very much a software engineer and not a hardware expert. However I am competent enough to be able to provide a minimum spec of what I would like my computer to behave if the power cable is pulled from it.

Re: The Long Goodbye

2012-08-06 10:36 • by Ben Jammin (unregistered)
It's that he's concerned with HR about:


"That's what 'night-shift' means," Greta replied with a smirk. "You were here for the whole thing last night."


But not concerned about hr/police about:


With the lights off on his side, Greta couldn't see him, but he could watch her through the glass walls.

Re: The Long Goodbye

2012-08-06 10:37 • by Ben Jammin (unregistered)
386466 in reply to 386465
funny*

My mind types faster than my fingers

Re: The Long Goodbye

2012-08-06 10:39 • by Rootbeer
386467 in reply to 386457
"I still think it's a WTF that desktop computers and servers are not like laptops with their own internal power source with the mains acting as just a recharger."

It's a solved problem for servers, as any competently designed server room will already have everything critical getting its power through a UPS. In the case in this story, it's possible that the server room proper has a UPS, but the overflow in the Copy Center does not.

It's kind of a solved problem for desktops, too, as home UPS units capable of gently shutting down a home computer and its peripherals in the event of an outage can be bought for under $100.

What I don't understand is why more server rooms don't let the UPSes deliver DC voltage direct to the server motherboards at all times.

Re: The Long Goodbye

2012-08-06 10:52 • by Infinite Time and Space (unregistered)
386469 in reply to 386424
ochrist:
I suggest they put up a note in the room with this text:

ACHTUNG!
ALLES TURISTEN UND NONTEKNISCHEN LOOKENPEEPERS!
DAS KOMPUTERMASCHINE IST NICHT FÜR DER GEFINGERPOKEN UND MITTENGRABEN! ODERWISE IST EASY TO SCHNAPPEN DER SPRINGENWERK, BLOWENFUSEN UND POPPENCORKEN MIT SPITZENSPARKSEN.
IST NICHT FÜR GEWERKEN BEI DUMMKOPFEN. DER RUBBERNECKEN SIGHTSEEREN KEEPEN DAS COTTONPICKEN HÄNDER IN DAS POCKETS MUSS.
ZO RELAXEN UND WATSCHEN DER BLINKENLICHTEN.

PS: What The Fr1st!


Two years of college German and that is about as well as I can speak it.

Re: The Long Goodbye

2012-08-06 11:01 • by D-Coder
386470 in reply to 386467
Rootbeer:
"I still think it's a WTF that desktop computers and servers are not like laptops with their own internal power source with the mains acting as just a recharger."

It's a solved problem for servers, as any competently designed server room...
...will not end up on this site.

Re: The Long Goodbye

2012-08-06 11:10 • by ufmace (unregistered)
First response to reading: the standard chuckle

Second response: Wait a second, how did they not figure that out sooner? What kind of troubleshooting plan never ruled out a loss of connectivity to the server? What kind of logging system never noted the type of error? While we're at it, what kind of manager tells their employees to not bother trying to fix a simple problem like that? Groan.

Re: The Long Goodbye

2012-08-06 11:13 • by Mikerad (unregistered)
386473 in reply to 386465
Ben Jammin:
It's that he's concerned with HR about:


"That's what 'night-shift' means," Greta replied with a smirk. "You were here for the whole thing last night."


But not concerned about hr/police about:


With the lights off on his side, Greta couldn't see him, but he could watch her through the glass walls.


Everyone knows HR doesn't exist after 5pm...

Re: The Long Goodbye

2012-08-06 11:14 • by EmptyJay (unregistered)
386474 in reply to 386442
Amen.

I can't stand not having built-in overhead lights. We installed them in both bedrooms before we moved into our current place, and I added recessed lights to the living room after about 2 years of suffering without.

I still find the switched outlets to be useful, but I'm slowly converting them to constant power.

Re: The Long Goodbye

2012-08-06 11:14 • by operagost
386475 in reply to 386445
Some Jerk:
unsafe if you have your fire detection plugged into that spot.

Pretty much handled by the NEC code that says you must have a dedicated circuit for fire detection and alarm equipment, and it can't have a switch on it. Try again.

Re: The Long Goodbye

2012-08-06 11:16 • by operagost
386476 in reply to 386451
foo:
Some Jerk:
unsafe if you have your fire detection plugged into that spot.
This, or something like: "Hmm, this socket is dead, I'll have to open it and check the wires. But let me just turn on the lights so I can see better ..."

Like when handling a firearm, always assume the circuit is live unless you've flipped the breaker and tested for current at the receptacle. Do they have untrained morons playing with electricity in the EU?

Re: The Long Goodbye

2012-08-06 11:22 • by foo (unregistered)
386477 in reply to 386464
Cbuttius:
dkf:
Cbuttius:
I still think it's a WTF that desktop computers and servers are not like laptops with their own internal power source with the mains acting as just a recharger.
It's called a UPS, and it's a feature of systems managed by someone who's actually competent.
Cbuttius:
Even if they were not able to hold power for a period of time (due to consuming far more energy than laptops) they could maintain power for a short period in order to be able to "hibernate" by persisting the current state and resume as soon as the main power source came back.
Assuming that they've been wired up right, they only need to hold power until the onsite generator comes up and stabilizes. Of course, experience of operations at a national datacenter tells me that the trick is to remember to put the network routers and AC on the generator too. And to remember to check that the upstream ISP has done the same thing…


I am not competent in that field, I am very much a software engineer and not a hardware expert. However I am competent enough to be able to provide a minimum spec of what I would like my computer to behave if the power cable is pulled from it.
As others said, you can have it. It's just a matter of cost. Many people (including myself :) don't spend the money, since power outages are very rare where I live, and journaled file systems avert most problems if they do occur.

However, note that this story was about the network switch losing power. Shutting it down gently wouldn't help a bit.

Re: The Long Goodbye

2012-08-06 11:22 • by Remy Porter
386478 in reply to 386476
I've done some wiring with the breaker on. It's the kind of mistake that you only make two or three times before you finally learn your lesson. Just call me "Sparky".

//Kidding- the only times I've gotten shocked from an outlet have been for far dumber reasons than not turning off the breaker.

Re: The Long Goodbye

2012-08-06 11:24 • by foo (unregistered)
386479 in reply to 386476
operagost:
foo:
Some Jerk:
unsafe if you have your fire detection plugged into that spot.
This, or something like: "Hmm, this socket is dead, I'll have to open it and check the wires. But let me just turn on the lights so I can see better ..."

Like when handling a firearm, always assume the circuit is live unless you've flipped the breaker and tested for current at the receptacle. Do they have untrained morons playing with electricity in the EU?
Sure. Don't they in the US (or anywhere else)? I mean people working on their own home's installations; sure, they're often stupid, and saftely regulations just add one layer of protection.

Re: The Long Goodbye

2012-08-06 11:24 • by Some Jerk (unregistered)
386480 in reply to 386475
operagost:
Some Jerk:
unsafe if you have your fire detection plugged into that spot.

Pretty much handled by the NEC code that says you must have a dedicated circuit for fire detection and alarm equipment, and it can't have a switch on it. Try again.


that was intended for humor dude. Sort of like charging the pacemaker. TRY AGAIN :p
« PrevPage 1 | Page 2 | Page 3Next »

Add Comment