- Feature Articles
- CodeSOD
- Error'd
- Forums
-
Other Articles
- Random Article
- Other Series
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
So, is TRWTF here that some unknown guys with blue overalls have unrestricted access to server room(s)?
Admin
Ok, all three of you are "frist" on a repeat. That's like being the frist guy to sleep with the hooker today.
captcha: abigo - the grade of the air conditioner repair men
Admin
Ok then... Blue collar workers work at 2am? I thought it was only us devs...
Captcha: genitus - who comes up with these?
Admin
And as for Phil, the developer who submitted this story, he got the day off...
Blah... I don't believe the end of the story. That never happens
Admin
Yes, I believe it is.
Admin
Ich bin Frist!
Admin
Did you mean: Ich bin fristgerecht gekündigt worden. ?
Admin
The Real TRWTF: "One summer day, at 2am"
Admin
Well, there's nothing in the story to suggest their access was unrestricted. Merely that the hero of the story hadn't called them.
Admin
I think there are better stories to promote to "classic" status. Why should the air-con guys even know (or care) about the cooling requirements of a server room? It's far more of a WTF when the perpetrators are the IT staff themselves and I know we've got a few of those in the archive. Still, thanks anyway.
Admin
The office was located in the arctic circle. Obviously.
Admin
I think the WTF is more along the lines of that their servers are so delicate that they can't handle a thermal shutdown with having to be rebuilt! I mean, really! My POS Exchange box can handle that!
Admin
I have no idea what you just said...
Captcha sagaciter.. what?
Admin
WhyTF "fristgerecht"?
Admin
"But the phone lines were dead, too, which could only mean one thing: a bomb, a fire, or a giant robot wreaking havoc throughout the city."
1 = 3
TRWTF
Admin
Of course, TRWTF is that IT didn't know the maintenance was going to be performed.
Admin
or != and TRRWTF
Admin
You win car anologies. I LOLed
Admin
Meh, I prefer the one where the drone turned off the AC to save energy and mailed everyone about it.
c even ba to find the URL
Admin
Admin
There are two (or three?) levels of redundant servers, but THEY'RE ALL IN THE SAME SERVER ROOM?
That's TRWTF right there. Anything bad happens (power out, flood, robbery, airplane crash, clueless HVAC repairmen) and you've got a single point of failure. I think the AC guys are blameless -- it's the guys who decided to put all their eggs in one refrigerated basket that deserve the blame.
Admin
I used to work at a large hospital and similar things happened several times for us. Blue collar workers let into the server room by themselves doing building maintenance work without anyone in the IT department even knowing about it beforehand.
Once we caught a stupid fuck drilling holes in the concrete walls without covering even the nearby servers. With an old, large, heavy power drill plugged into a power outlet inside a server rack. Which killed several critical servers.
Another time they were replacing the card reader lock of the server room door and left the door open and unguarded for several hours in the middle of the day. By mere luck a coworker who was going there to change a backup tape caught a cleaner who were just about to empty a bucket full of soap water on the floor...
I don't think that things like these are too uncommon unfortunately.
Admin
I think someone forgot how to count to ONE.
Admin
Admin
Exactly what I was thinking.
And the other thing about all these aircon-fail stories: Even regular consumer grade PCs have the ability to shutdown the system in case the CPU overheats - so all that should be necessary after an incident like this is rebooting all the servers - which is bad in itself of course for high availability systems, but never does take 36 hours...
Admin
I remember when this site had TWO NEW STORIES every day; now it's one article per day and full of filler like shitty comics, useless programming problems outside of the scope of the site, mandatory ad day, supposed WTF's that aren't really all that bad (e.g. unencrypted passwords), and Alex's soapbox (barf).
Get back to your roots, bring the WTF stories.
Admin
That's one way to save on cooling costs.
Admin
Then it should say one of three things not one thing. Or better:
Now the "one thing" is a disaster, and that one disaster could be any number of things. This was obviously the intent of the sentence.
Admin
Because Giant Robots never cause fires or drop bombs?
Admin
CAPTCHA: damnum I knew it!
Admin
Can servers overheat there? Global warming is worse than I thought.
Admin
Global warming is TRWTF.
Admin
And the electrician does electricty. Who cares what it's being used for, just yank the big breaker so he can work on it safely.
In truth, it might not matter what it's used for, but it does matter if it's in use when you want to fix it. To go with the car analogy, a car mechanic can't force you to pull over and get out so he can fix your car while you're driving it, you have to bring it in to him.
Admin
It's the two repair guys from the movie "Brazil"! See http://www.youtube.com/watch?v=VNFuySgwQ30, starting at 2:57. The guy should have just asked if they had a 27B/6, and they would have left.
Admin
If the AC guys are contracted to keep the AC units maintained and running in a server room then they damn well better understand that having that room cold is critical to keeping those servers running. And if they don't it's time for the business to find some new contractors who have the ability to think.
Admin
As much as I enjoyed the article and agree with most of the comments (except for the bit about the mechanic), I have to admit a dirty little secret: our server rack is open and placed just outside the toilets. A camera does cover the tack, but the camera server is in the same rack.
Some more confessions: Of the 8 or so server in-house, most of them run Windows XP. The domain controller doesn't have any anti-virus software running on it. Many of the machines we have at clients are running fedora 5. (I don't think this is too much of a problem, except they are a default desktop install, even though their purpose is to stream video) The same root password is used almost everywhere, and is given to most of the employees.
I've only been here a couple of months, and am a dev, with a bit of . I have so much work to do...
Admin
Admin
I once had something similar: The air output of an AC unti broke down, so the unit continued to cool the same air over and over, until the water condensed on the unit...
We noticed temperatures rising, and called the 'blue-collar'. He removed the uunit's cover, and we were looking at a solifd lump of ice hanging on the wall at a room temperature of 35C (~95F I believe). He pushed the on/off switch and the next day the ice should be gone.
Except that it wasn't. Turned out that the on/off switch itself was frozen stuck too... Unplugging the unit had the desired effect...
Admin
TRWTF is that Phil got the day off while Mark got fucked. You guys couldn't lend a hand?
Admin
Disagreed. An adequate server room will have redundant power supply (usually two independent lines plus a generator). Natural disaster and stupidity can of course only be met by adding a layer of geographically independent redundancy, but otherwise it is perfectly legitimate to set up redundant systems at a single location (note, the room does have redundant AC).
If they were gunning for three-nines uptime, they should have had another site, but for (say) 99.5% p.a., it's not required (while redundant servers are).
Admin
That's an awful funny definition of "one".
Admin
WTF - they shouldn't even be in the same Time Zone
Admin
Admin
Admin
It's the opposite of "fristlos".
Admin
Sounds like a mickey-mouse operation. They need a redundant datacenter (or 2, or 3).
Admin
Well, obviously yes. Otherwise they wouldn't have had the AC people out. Or are you questioning Alex's infallibility and suggesting he made a mistake when he called 2 am "a summer day"? Blasphemer!
Admin
I was thinking that too. These stories all end with "and everything was fried!". Is that some major embellishment, or wouldn't the servers just all shut down?
Admin
That afternoon, while I was gone, one of the VP's had something he wanted me to do for him. But when he walked into my office, and saw it all nice and neat and me not there, he went to the department head's office and asked why I had quit.
Admin