Comment On Classic WTF: Keepin' It Cool

So, is TRWTF here that some unknown guys with blue overalls have unrestricted access to server room(s)?

Ok, all three of you are "frist" on a repeat. That's like being the frist guy to sleep with the hooker today.

captcha: abigo - the grade of the air conditioner repair men

Ok then... Blue collar workers work at 2am? I thought it was only us devs...

Captcha: genitus - who comes up with these?

And as for Phil, the developer who submitted this story, he got the day off...

Blah... I don't believe the end of the story. That never happens

G:

So, is TRWTF here that some unknown guys with blue overalls have unrestricted access to server room(s)?

You mean the large bank server room where massive amounts of money are changing hands daily?

Yes, I believe it is.

Ich bin Frist!

Fredrik:

Ich bin Frist!

Did you mean: Ich bin fristgerecht gekündigt worden. ?

The Real TRWTF:
"One summer day, at 2am"

G:

So, is TRWTF here that some unknown guys with blue overalls have unrestricted access to server room(s)?

Well, there's nothing in the story to suggest their access was unrestricted. Merely that the hero of the story hadn't called them.

I think there are better stories to promote to "classic" status. Why should the air-con guys even know (or care) about the cooling requirements of a server room? It's far more of a WTF when the perpetrators are the IT staff themselves and I know we've got a few of those in the archive. Still, thanks anyway.

@Deprecated:

The Real TRWTF:
"One summer day, at 2am"

The office was located in the arctic circle. Obviously.

I think the WTF is more along the lines of that their servers are so delicate that they can't handle a thermal shutdown with having to be rebuilt! I mean, really! My POS Exchange box can handle that!

I have no idea what you just said...




Captcha sagaciter.. what?

derula:

Fredrik:

Ich bin Frist!

Did you mean: Ich bin fristgerecht gekündigt worden. ?

WhyTF "fristgerecht"?

"But the phone lines were dead, too, which could only mean one thing: a bomb, a fire, or a giant robot wreaking havoc throughout the city."

1 = 3

TRWTF

Anonymous:

I think there are better stories to promote to "classic" status. Why should the air-con guys even know (or care) about the cooling requirements of a server room?

That's like an auto mechanic deciding to remove the seats from your car unannounced because he's unaware of the seating requirements of your butt.

Of course, TRWTF is that IT didn't know the maintenance was going to be performed.

BBT:

"But the phone lines were dead, too, which could only mean one thing: a bomb, a fire, or a giant robot wreaking havoc throughout the city."

1 = 3

TRWTF

or != and
TRRWTF

operagost:

That's like an auto mechanic deciding to remove the seats from your car unannounced because he's unaware of the seating requirements of your butt.

You win car anologies. I LOLed

Meh, I prefer the one where the drone turned off the AC to save energy and mailed everyone about it.

c even ba to find the URL

operagost:

Anonymous:

I think there are better stories to promote to "classic" status. Why should the air-con guys even know (or care) about the cooling requirements of a server room?

That's like an auto mechanic deciding to remove the seats from your car unannounced because he's unaware of the seating requirements of your butt.

Of course, TRWTF is that IT didn't know the maintenance was going to be performed.

I disagree. Air-con guys do air-con - it is not their responsibility to worry about what the air-con is being used for. In the same way that your mechanic doesn't need to know where or how you drive your car in order to fix it. He doesn't care where you drive it, it's none of his business, it doesn't affect how he fixes your car.

There are two (or three?) levels of redundant servers, but THEY'RE ALL IN THE SAME SERVER ROOM?

That's TRWTF right there. Anything bad happens (power out, flood, robbery, airplane crash, clueless HVAC repairmen) and you've got a single point of failure. I think the AC guys are blameless -- it's the guys who decided to put all their eggs in one refrigerated basket that deserve the blame.

I used to work at a large hospital and similar things happened several times for us. Blue collar workers let into the server room by themselves doing building maintenance work without anyone in the IT department even knowing about it beforehand.

Once we caught a stupid fuck drilling holes in the concrete walls without covering even the nearby servers. With an old, large, heavy power drill plugged into a power outlet inside a server rack. Which killed several critical servers.

Another time they were replacing the card reader lock of the server room door and left the door open and unguarded for several hours in the middle of the day. By mere luck a coworker who was going there to change a backup tape caught a cleaner who were just about to empty a bucket full of soap water on the floor...

I don't think that things like these are too uncommon unfortunately.

Retarticle:

which could only mean one thing: a bomb, a fire, or a giant robot wreaking havoc throughout the city.

I think someone forgot how to count to ONE.

Anon Ymous:

Retarticle:

which could only mean one thing: a bomb, a fire, or a giant robot wreaking havoc throughout the city.

I think someone forgot how to count to ONE.

You seriously think that all three of those unlikely events might be happening simultaneously? I think it's far more likely to be one of those things rather than all of them.

OMG:

There are two (or three?) levels of redundant servers, but THEY'RE ALL IN THE SAME SERVER ROOM?

That's TRWTF right there. Anything bad happens (power out, flood, robbery, airplane crash, clueless HVAC repairmen) and you've got a single point of failure. I think the AC guys are blameless -- it's the guys who decided to put all their eggs in one refrigerated basket that deserve the blame.

Exactly what I was thinking.

And the other thing about all these aircon-fail stories:
Even regular consumer grade PCs have the ability to shutdown the system in case the CPU overheats - so all that should be necessary after an incident like this is rebooting all the servers - which is bad in itself of course for high availability systems, but never does take 36 hours...

I remember when this site had TWO NEW STORIES every day; now it's one article per day and full of filler like shitty comics, useless programming problems outside of the scope of the site, mandatory ad day, supposed WTF's that aren't really all that bad (e.g. unencrypted passwords), and Alex's soapbox (barf).

Get back to your roots, bring the WTF stories.

That's one way to save on cooling costs.

Anonymous:

Anon Ymous:

Retarticle:

which could only mean one thing: a bomb, a fire, or a giant robot wreaking havoc throughout the city.

I think someone forgot how to count to ONE.

You seriously think that all three of those unlikely events might be happening simultaneously? I think it's far more likely to be one of those things rather than all of them.

Then it should say one of three things not one thing. Or better:

which could only mean one thing a disaster such as: a bomb, a fire, or a giant robot...

Now the "one thing" is a disaster, and that one disaster could be any number of things. This was obviously the intent of the sentence.

Anonymous:

Anon Ymous:

Retarticle:

which could only mean one thing: a bomb, a fire, or a giant robot wreaking havoc throughout the city.

I think someone forgot how to count to ONE.

You seriously think that all three of those unlikely events might be happening simultaneously? I think it's far more likely to be one of those things rather than all of them.

Because Giant Robots never cause fires or drop bombs?

Jaco:

Ok then... Blue collar workers work at 2am? I thought it was only us devs...

Captcha: genitus - who comes up with these?

Maybe it's the Automated Curse Generator

CAPTCHA: damnum
I knew it!

Anon:

@Deprecated:

The Real TRWTF:
"One summer day, at 2am"

The office was located in the arctic circle. Obviously.

Can servers overheat there?
Global warming is worse than I thought.

!?:

Anon:

@Deprecated:

The Real TRWTF:
"One summer day, at 2am"

The office was located in the arctic circle. Obviously.

Can servers overheat there?
Global warming is worse than I thought.

Global warming is TRWTF.

Anonymous:

operagost:

Anonymous:

I think there are better stories to promote to "classic" status. Why should the air-con guys even know (or care) about the cooling requirements of a server room?

That's like an auto mechanic deciding to remove the seats from your car unannounced because he's unaware of the seating requirements of your butt.

Of course, TRWTF is that IT didn't know the maintenance was going to be performed.

I disagree. Air-con guys do air-con - it is not their responsibility to worry about what the air-con is being used for. In the same way that your mechanic doesn't need to know where or how you drive your car in order to fix it. He doesn't care where you drive it, it's none of his business, it doesn't affect how he fixes your car.

And the electrician does electricty. Who cares what it's being used for, just yank the big breaker so he can work on it safely.

In truth, it might not matter what it's used for, but it *does* matter if it's in use when you want to fix it. To go with the car analogy, a car mechanic can't force you to pull over and get out so he can fix your car while you're driving it, you have to bring it in to him.

It's the two repair guys from the movie "Brazil"!
See http://www.youtube.com/watch?v=VNFuySgwQ30, starting at 2:57.
The guy should have just asked if they had a 27B/6, and they would have left.

Anonymous:

operagost:

Anonymous:

I think there are better stories to promote to "classic" status. Why should the air-con guys even know (or care) about the cooling requirements of a server room?

That's like an auto mechanic deciding to remove the seats from your car unannounced because he's unaware of the seating requirements of your butt.

Of course, TRWTF is that IT didn't know the maintenance was going to be performed.

I disagree. Air-con guys do air-con - it is not their responsibility to worry about what the air-con is being used for. In the same way that your mechanic doesn't need to know where or how you drive your car in order to fix it. He doesn't care where you drive it, it's none of his business, it doesn't affect how he fixes your car.

If the AC guys are contracted to keep the AC units maintained and running in a server room then they damn well better understand that having that room cold is critical to keeping those servers running. And if they don't it's time for the business to find some new contractors who have the ability to think.

As much as I enjoyed the article and agree with most of the comments (except for the bit about the mechanic), I have to admit a dirty little secret: our server rack is open and placed just outside the toilets. A camera does cover the tack, but the camera server is in the same rack.

Some more confessions:
Of the 8 or so server in-house, most of them run Windows XP.
The domain controller doesn't have any anti-virus software running on it.
Many of the machines we have at clients are running fedora 5. (I don't think this is too much of a problem, except they are a default desktop install, even though their purpose is to stream video)
The same root password is used almost everywhere, and is given to most of the employees.

I've only been here a couple of months, and am a dev, with a bit of . I have so much work to do...

!?:

Can servers overheat [in the Arctic Circle]?

In the summer when it's also known as the land of the midnight sun? Sure. If you get a spell of good weather it can get really quite warm...

I once had something similar: The air output of an AC unti broke down, so the unit continued to cool the same air over and over, until the water condensed on the unit...

We noticed temperatures rising, and called the 'blue-collar'. He removed the uunit's cover, and we were looking at a solifd lump of ice hanging on the wall at a room temperature of 35C (~95F I believe). He pushed the on/off switch and the next day the ice should be gone.

Except that it wasn't. Turned out that the on/off switch itself was frozen stuck too... Unplugging the unit had the desired effect...

TRWTF is that Phil got the day off while Mark got fucked. You guys couldn't lend a hand?

OMG:

There are two (or three?) levels of redundant servers, but THEY'RE ALL IN THE SAME SERVER ROOM?

That's TRWTF right there. Anything bad happens (power out, flood, robbery, airplane crash, clueless HVAC repairmen) and you've got a single point of failure. I think the AC guys are blameless -- it's the guys who decided to put all their eggs in one refrigerated basket that deserve the blame.

Disagreed. An adequate server room will have redundant power supply (usually two independent lines plus a generator). Natural disaster and stupidity can of course only be met by adding a layer of geographically independent redundancy, but otherwise it is perfectly legitimate to set up redundant systems at a single location (note, the room does have redundant AC).

If they were gunning for three-nines uptime, they should have had another site, but for (say) 99.5% p.a., it's not required (while redundant servers are).

which could only mean one thing: a bomb, a fire, or a giant robot wreaking havoc throughout the city.

That's an awful funny definition of "one".

OMG:

There are two (or three?) levels of redundant servers, but THEY'RE ALL IN THE SAME SERVER ROOM?

WTF - they shouldn't even be in the same Time Zone

Ben:

Anonymous:

operagost:

Anonymous:

I think there are better stories to promote to "classic" status. Why should the air-con guys even know (or care) about the cooling requirements of a server room?

That's like an auto mechanic deciding to remove the seats from your car unannounced because he's unaware of the seating requirements of your butt.

Of course, TRWTF is that IT didn't know the maintenance was going to be performed.

I disagree. Air-con guys do air-con - it is not their responsibility to worry about what the air-con is being used for. In the same way that your mechanic doesn't need to know where or how you drive your car in order to fix it. He doesn't care where you drive it, it's none of his business, it doesn't affect how he fixes your car.

And the electrician does electricty. Who cares what it's being used for, just yank the big breaker so he can work on it safely.

In truth, it might not matter what it's used for, but it *does* matter if it's in use when you want to fix it. To go with the car analogy, a car mechanic can't force you to pull over and get out so he can fix your car while you're driving it, you have to bring it in to him.

But if the air-con guys are just thinking air-con is for people, doing it at 2am avoids causing anyone to overheat. They probably had a work order to do the chiller-bars/mainteneance on AC unit 1 and figured they'd do the second while they were there so when the work order for unit 2 came down, they could chill for the night.

Robo:

If the AC guys are contracted to keep the AC units maintained and running in a server room...
<snip>

It never said this anywhere in the article. You are making a huge assumption that they were contracted by this company for this specific job. For all you know, they were called out for an ad-hoc fix and have come from an air-con company that the bank had previously never used. You can't just make up the bits of the story that weren't mentioned.

grzlbrmft:

derula:

Fredrik:

Ich bin Frist!

Did you mean: Ich bin fristgerecht gekündigt worden. ?

WhyTF "fristgerecht"?

It's the opposite of "fristlos".

Uhhh:

I think the WTF is more along the lines of that their servers are so delicate that they can't handle a thermal shutdown with having to be rebuilt! I mean, really! My POS Exchange box can handle that!

Sounds like a mickey-mouse operation. They need a redundant datacenter (or 2, or 3).

!?:

Anon:

@Deprecated:

The Real TRWTF:
"One summer day, at 2am"

The office was located in the arctic circle. Obviously.

Can servers overheat there?
Global warming is worse than I thought.

Well, obviously yes. Otherwise they wouldn't have had the AC people out. Or are you questioning Alex's infallibility and suggesting he made a mistake when he called 2 am "a summer day"?
Blasphemer!

methinks:

And the other thing about all these aircon-fail stories:
Even regular consumer grade PCs have the ability to shutdown the system in case the CPU overheats - so all that should be necessary after an incident like this is rebooting all the servers - which is bad in itself of course for high availability systems, but never does take 36 hours...

I was thinking that too. These stories all end with "and everything was fried!". Is that some major embellishment, or wouldn't the servers just all shut down?

Lee K-T:

And as for Phil, the developer who submitted this story, he got the day off...

Blah... I don't believe the end of the story. That never happens

In the early 1980s, our PDP-11 was down for a day once and I took most of the day off after cleaning up my office first.

That afternoon, while I was gone, one of the VP's had something he wanted me to do for him. But when he walked into my office, and saw it all nice and neat and me not there, he went to the department head's office and asked why I had quit.

Teeyouecksayee:

a coworker who was going there to change a backup tape caught a cleaner who were just about to empty a bucket full of soap water on the floor...

I know techies are soapdodgers, but some reach the epic levels of vampire-vs.-holy-water aversion. "Who cares my servers have been unguarded for hours, as long as the floor isn't cleaned."