Comment On CSI: Server Room

Somehow I knew it was going to be a light switch problem. We've seen a lot of these lately.

So yeah, this isn't the FRIST time I've seen this.

I was expecting this to involve a vacuum cleaner again.

Reminds me of the UL (?) about the server crash that was traced to the cleaning crew unplugging the machine to plug in their vacuum cleaner after-hours.

What are you doing?
I just turned the kitchen light off.

That seems like an unlikely answer to that question. Something like "getting ready to eat my breakfast bitch" seems more likely.

I was expecting some kind of automatic / motion sensing light switch. But a manual one! ... that is a WTF

(Captcha: Ninjas)

I once worked in a university radioisotope lab where the cleaners unplugged a small fridge so they could use the outlet. They never plugged the fridge back in and when we showed up in the morning the ice in the fridge had melted and fallen on some beakers and broken them. Thankfully it wasn't anything radioactive it was liquid methyl methacrylate (plexiglass monomer). The smell was horrible.

Some years ago (ok ok - MANY years ago) I worked for a computer company (the clue is in my name).

We had a pretty large setup, with a bloody nice computer room - AC, UPS all the good stuff one would accept.

So, it came to pass that we needed an extra VS10000 in the computer room but our electrician told us that it would cost a certain sum of money to run a new electrical circuit into the computer room to handle the load. However, our Finance Director decided that his brother could do the job for considerably less than the quote from the qualified sparks.

A couple of weeks after the new VS was installed, I was in the coffee station (with a murderous hangover). I had filled my cup with the brew, came out of the coffee station and tripped over a floor socket cover that had suddenly appeared. Coffee spilled. The lights in the computer room remained on, the older servers we had there continued working, but the NEW! IMPROVED! VS died.

Upon investigation, it was discovered that, instead of doing expensive and sensible electrical stuff, the FD's brother decided to power the new VS off the one available power socket. In the floor. Outside the computer room. Not on UPS.

Even worse. Because the cover wouldn't close properly, he removed the plug off the power strip he had run to this socket, and just put the wires directly into the terminals. In the floor. Outside the Coffee station.

ElRobbo:

Reminds me of the UL (?) about the server crash that was traced to the cleaning crew unplugging the machine to plug in their vacuum cleaner after-hours.

I heard another version - about a janitor unplugging life support systems in an ICU for the same purpose.

HOLY SHIT, CANYON.MID. Wow, that is a throwback. I feel like im having acid flashbacks, but way more boring.

Routine WTF - but nice story Jake!

The real WTF is that there are some computer geeks out there who don't work in total darkness!!!

Soviut:

The real WTF is that there are some computer geeks out there who don't work in total darkness!!!

Hmmm - I ordered a Dell XPS 720 3+ weeks ago and they still haven't shipped it - at home, my computer desk is in total darkness - does that count?

That depends... Are you working?

This is like the problem that plagued one spanking-new TGV trainset when it went in service, back in 1989. It would run fine until it would go in emergency braking mode.

when it happenned a few times, they took it out of service, but could not find anything wrong.

So they put it back in service, until, days or weeks later, it would pull the same stunt. Back to the shop, inspect, and find nothing.

Eventually, a senior maintenance tech would ride the trainset until it would “crash” likewise. And in the few times it crashed, the tech would not find anything. That is until his boss rode with him, and went to the bathroom.

A first-class bathroom, as it happens, since bosses have first-class passes, and not lowly peons.

Just as the boss got into the bathroom and locked the door, he sensed that the train immediately got into emergency.

He hopped quickly hopped on the intercom and buzzed the enginner: “what did you just do?”

— Well, I just went into rheostatic braking (a mode where the train’s motors are run as generators, and the power is dissipated through roof-mounted resistors)…

So, at least, they had a clue. They managed to make it to destination, and following some hands-on diagnostics, they managed to pinpoint it to a wiring error that would trigger an emergency pneumatic braking whenever the train was in rheostatic mode (which is about 5% of the time it runs).

> Nick insisted on a trinagulated backup plan

is that the one where you have 3 people nagging you about it?

They received a 1Tb NAS drive...

Not an urban legend. I had a client at a previous job who's core system was mysteriously going down between 9 and 9:30 PM just about every 2nd or 3rd Friday, long after everyone had gone home.

Eventually the boss got sick of paying overtime for all the after-hours support time and convinced the client to have someone stay late. Lo and behold, the cleaning crew came in right around nine and proceeded to unplug the server and plug in a vacuum. The UPS alarm went off, but the cleaning crew ignored it, vacuumed away, then unplugged the vacuum and plugged the server back in. Depending on whether the UPS could last, the server might or might not go down.

We actually updated our operations manuals to include a new section on 'Educating Facilities Management', along with instructions on connecting the UPS to the server so that, among other things, the server could log an event and send out a page if the UPS lost AC power.

I would be pissed!

Once again we note that by observing the event, we have altered the outcome.

/Heisenberg nods approvingly.

Floor Buffer - Wednesday Nights - Small company called Astrosystems, in Lake Success NY - I'm the poor guy who had to go plug it back in every Thursday AM - so no UL - the company did have a Micro-vax in it's own room, and that stayed up - it was just the Owners Assistant's machine - and she was too dumb to figure it out, and plug it in herself

Damn, right. I haven't heard that in over 10 years. And whoever created the MP3 must have deliberately found the soundcard with the crappiest FM chip for playback.

I'm gonna have to find MINUET.MID....

I installed a UNIX server at a small academic college's library in the early '90s. The library also had a few PCs for students to use. The cleaning staff had been instructed to turn off all of the 'computers' every night. (This was in the Windows 3.x days, so shutting them down wasn't necessary; they just hit the power buttons.) When it was realised that the cleaning staff were hitting the power button on the server every night, they were asked to stop turning off the computers. They politely declined. It took several days and considerable paperwork to change their standing instructions and, in the meantime, they conscienciously continued to do exactly what their job description said to do.

I could see the response being "Nothing, I just turned the light off" if the question was something like "Our server just failed, what did you just do?"

Again, how is this Worse Than Failure?

Seems like a pretty decent job of sleuthing out the problem, and solving it.

Isn't Worse Than Failure supposed to be something about knowing it sucks and then continuing on in the same direction, or stories of gross incompetence?

snoofle:

Soviut:

The real WTF is that there are some computer geeks out there who don't work in total darkness!!!

Hmmm - I ordered a Dell XPS 720 3+ weeks ago and they still haven't shipped it - at home, my computer desk is in total darkness - does that count?

The WTF is buying a DELL.

unklegwar:

Again, how is this Worse Than Failure?

Seems like a pretty decent job of sleuthing out the problem, and solving it.

Isn't Worse Than Failure supposed to be something about knowing it sucks and then continuing on in the same direction, or stories of gross incompetence?

Wiring server power to the same circuit as the kitchen light isn't considered gross incompetence?

captcha=quake

unklegwar:

Again, how is this Worse Than Failure?

Seems like a pretty decent job of sleuthing out the problem, and solving it.

Isn't Worse Than Failure supposed to be something about knowing it sucks and then continuing on in the same direction, or stories of gross incompetence?

The WTF (in the original sense) is in the problem's inconvenient and seemingly perverse habit of occurring only when no one was looking. Clueing us in as to the Heisenbergian nature of the situation adds to the amusement.

bruzie:

Damn, right. I haven't heard that in over 10 years. And whoever created the MP3 must have deliberately found the soundcard with the crappiest FM chip for playback.

I'm gonna have to find MINUET.MID....

That's what CANYON.MID always sounded like on my computer. But the question is, why did they make it into an MP3 for that page? Why not just embed the original MIDI file?

unklegwar:

The WTF is buying a DELL.

Your wit is like unto a razor. I almost laughed. Well, not really.

Well written, but disappointingly predictable.
There was no mystery or intrigue in reading the post.

masonReloaded:

unklegwar:

Again, how is this Worse Than Failure?

Seems like a pretty decent job of sleuthing out the problem, and solving it.

Isn't Worse Than Failure supposed to be something about knowing it sucks and then continuing on in the same direction, or stories of gross incompetence?

Wiring server power to the same circuit as the kitchen light isn't considered gross incompetence?

No. The incompetence is simply in wiring it on the wrong side of the switch.

Ex-wanger:

Some years ago (ok ok - MANY years ago) I worked for a computer company (the clue is in my name).

We had a pretty large setup, with a bloody nice computer room - AC, UPS all the good stuff one would accept.

So, it came to pass that we needed an extra VS10000 in the computer room but our electrician told us that it would cost a certain sum of money to run a new electrical circuit into the computer room to handle the load. However, our Finance Director decided that his brother could do the job for considerably less than the quote from the qualified sparks.

A couple of weeks after the new VS was installed, I was in the coffee station (with a murderous hangover). I had filled my cup with the brew, came out of the coffee station and tripped over a floor socket cover that had suddenly appeared. Coffee spilled. The lights in the computer room remained on, the older servers we had there continued working, but the NEW! IMPROVED! VS died.

Upon investigation, it was discovered that, instead of doing expensive and sensible electrical stuff, the FD's brother decided to power the new VS off the one available power socket. In the floor. Outside the computer room. Not on UPS.

Even worse. Because the cover wouldn't close properly, he removed the plug off the power strip he had run to this socket, and just put the wires directly into the terminals. In the floor. Outside the Coffee station.

No story that contains "my <insert relative here> can do that for a lot less" ever turns out good.

akatherder:

What are you doing?
I just turned the kitchen light off.

That seems like an unlikely answer to that question. Something like "getting ready to eat my breakfast bitch" seems more likely.

I've never eaten a breakfast bitch before.

Ex-wanger:

Some years ago (ok ok - MANY years ago) I worked for a computer company (the clue is in my name).

We had a pretty large setup, with a bloody nice computer room - AC, UPS all the good stuff one would accept.

So, it came to pass that we needed an extra VS10000 in the computer room but our electrician told us that it would cost a certain sum of money to run a new electrical circuit into the computer room to handle the load. However, our Finance Director decided that his brother could do the job for considerably less than the quote from the qualified sparks.

Surely this must have been after An Wang had passed away. The Doctor was somewhat miserly but even so I cannot imagine anyone daring anything like this while he was still around.

-Harrow.

When the problem involves an issue on off hours, and it's not reproducible when employees are there, always check your electrical system.

This is a pretty obvious problem that if it took them more than a week to solve they should be ashamed.

Especially with all the urban legends about "unplugging the patient to plug in a vacuum" that cleaning staff wouldn't just randomly unplug something. Find an empty socket and use an extension cord!

Not to mention that trying to do the electrical on the cheap resulted in overtime costs while they tried to solve the problem. One of those "one step forward, two steps back" sort of situations. Likewise, the exiting illegal electrical would need to be fixed and cost even more money, unless they didn't mind the risk of spilling coffee on exposed leads, or tripping over them constantly.

Franz Kafka:

> Nick insisted on a trinagulated backup plan

is that the one where you have 3 people nagging you about it?

You beat me to that quote, damn you!

Am I the only one who had to look up trinagulate to make sure it was a typo?

Herohtar:

bruzie:

Damn, right. I haven't heard that in over 10 years. And whoever created the MP3 must have deliberately found the soundcard with the crappiest FM chip for playback.

I'm gonna have to find MINUET.MID....

That's what CANYON.MID always sounded like on my computer. But the question is, why did they make it into an MP3 for that page? Why not just embed the original MIDI file?

Because canyon.mid on today's soundcards sounds rather crap. ;p

BTW, here's an alternate rendition I made that probably sounds closer to how most people remember it.

ParkinT:

masonReloaded:

Wiring server power to the same circuit as the kitchen light isn't considered gross incompetence?

No. The incompetence is simply in wiring it on the wrong side of the switch.

Actually, sometimes it's a good thing. When my parents renovated their house, Dad did all the wiring (being an electronics tech at the local Synchrotron lab). He put in hard-wired 120v smoke detectors everywhere.

When the building inspector went through, that smoke detector wiring was the only thing he had a comment on... He required that at least one commonly used light be placed on the same circuit. That way, you'd know if there was a problem with the detectors if the light didn't come on.

It wasn't just nagulated once, but THREE TIMES! It was trinagulated. WTF?

ElRobbo:

Reminds me of the UL (?) about the server crash that was traced to the cleaning crew unplugging the machine to plug in their vacuum cleaner after-hours.

I have know this to happen, and seen the server with my own eyes and received 1st hand accounts from the sysadmin (Whom is now my co-worker after I moved departments). The server was not physically located in the server room, due to space constraints.

This leads me to believe that this sort of thing happens quite often, which is why people think it is UL: the details change every time someone tells it.

During the course of his investigation, did he finally find the UPS still packed in original box in which he was supposed to plug the NAS drive?

"Oh! *slap forehead* the UPS."

There's a freezer in the lab where I work with a friendly, helpful information sign on the front:

EVERYTHING IN THIS FREEZER CAN KILL YOU.

"Everything" in this case includes stuff like HF in aqueous solution and a variety of other solvents. For the chemically-illiterate, HF isn't quite as corrosive as the fictional acid blood of the creatures in the "Alien" movies, but it's pretty close.

It's properly contained within the freezer, so if the power goes out in theory nothing bad happens (but just in case it does, we make sure the power stays on ;-). Keeping the stuff solid makes it safer to handle--it's much less reactive as a solid, which is a good thing if some gets accidentally spilled or dropped.

Someguy:

akatherder:

What are you doing?
I just turned the kitchen light off.

That seems like an unlikely answer to that question. Something like "getting ready to eat my breakfast bitch" seems more likely.

I've never eaten a breakfast bitch before.

eh, they've not bad if you go for the ones with the ho-speed label on them.

man, how did I trudge up that lame wtf from my brain.. and I apologize for doing so.

Dan:

I feel like im having acid flashbacks, but way more boring.

Now that goes in the quotes file!

PeriSoft:

They received a 1Tb NAS drive...

What, we've given up on disrespecting the girlies now and have moved on to another well-known and despised minority? (I refer, of course, to Yankee fans.)

Incidentally, I don't really want to know why he's ripped off his penis; but why is he holding it in his hands?

Someguy:

I've never eaten a breakfast bitch before.

It's a lot like a breakfast burrito, but really more like a taco.

True story:

We just bought a new house. Cable guy comes to install the cable modem, but their is no power in the basement. The basement, and all exterior sockets, are on the GFI switch in the garage. I replaced the GFI, no change.

Being a new house, I call the builder, and they send out an electrician. He checks, and all sockets are working.

2 weeks later, I try to move the cable modem back down to the basement. No power.

Again, electrician comes out, and there is power running to all the sockets.

2 weeks later, same situation. I have to yell and scream, as the electrician didn't want to come out again. He did, and there was no power to the outside switch.

Turns out, it was wired after the exterior lights. As long as the lights were on, then we got power, like when the electricians came earlier. If you turned off the lights , then no power.

CAPTCHA Test: Tastey.. what I would have been if I tried to wire it myself.