- Feature Articles
- CodeSOD
- Error'd
-
Forums
-
Other Articles
- Random Article
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
I always like to throw in little extras for the folks that pay attention.
Admin
I don't think Pete is saying that Ryan should have actually done the remapping or moved a sensor himself, he's saying he should have contacted the HVAC people to fix it.
In one respect, that's reasonable. However, it ignores the fact that this was a company that was in the final stagger of it's death spiral, and nobody would be willing to pay the HVAC company to fix something like that when they're about to terminate occupancy!
Admin
Are modern datacenters actually controlled by HVAC zones? And have heat for that matter? The datacenters I have worked in all have had monster cooling units in the same room that perpetually pumped cold air into the raised floor. Working in them during the winter was always fun...
Admin
No, TRWTF is that several minutes later he was still playing the same game of MineSweeper. To set a record, he should have been done much sooner.
Admin
Reminds me of my first co-op job. This mid-sized company had a single server room: a closet off the main hallway. No, I mean a closet. The room was originally meant for an alternate bathroom, but they turned it into the company's data centre. It housed the PAX, Voice Mail, hold music CD player, Netware server running EVERYthing, network switches and the company's brand-new Internet connection / SMTP gateway.
In Summer, the HVAC would be running full blast into that room, but it still felt like a sauna working in there.
Admin
Admin
Speaking of Lone IT guys and HVAC, I have a similiar story. After a few rounds of layoffs and people seeking the employment higher ground I was the sole IT guy for a small company.
Not being the biggest company in the world we kept all the day-to-day servers in what amounted to a large closet. To be fair, it was a tricked out closet with plenty of power and it's own AC. Normally this was a pretty good setup and worked flawlessly.
That is until I opened the door one morning and was blasted by a heat wave. I later checked the temp sensor log and discovered it was around 120 degrees!! That is fine for outside in Phoenix during the summer but not in my server room.
Turns out that the AC had a single control line that determined wether to blow cold or hot air. Apply current on the line and it cools, remove the current and it provides heat. Well, that cable broke and turned my AC into a heater.
All told we only lost part of an UPS. It was a miracle. Like the story above, I also left shortly there after.
Admin
Alternate ending...
After he figured out the real problem, he implemented a stop gap measure until the building HVAC people could move the sensor.
He turned off the portable A/C unit, propped open the super secure server room door with a box of printer paper, and put a small, oscillating fan in the doorway blowing air into the NOC.
Within 5 minutes the situation was under control and he resumed his mine sweeper game. Weeks later, he closed the server room door, unplugged the still oscillating fan, and took the fan with him as he left the building for the last time.
Admin
OKey, you are bored silly with absolutely no work.
A server goes down. That has to be the event of the month, and that gets ignored?
The only friggen thing he had to do was to keep those up! Lazy lazy lazy (but I'm sure being a chair warmer breeds lazy).
Admin
Heh. I work at a place with a similar problem: the server room is still connected to the main buildings HVAC, in addition to having it's own dedicated cooling.
So in the summer, you turn down the thermostat and open the vents and let the extra AC from the main system augment the dedicated system.
But woe betide if you forget and leave them open until the winter, when the heat kicks in, because the thermostats are digital, and reset from a central location. So it switches to heat, notices that room A123 is FREEZING, and starts pumping in heat.
Admin
Can anyone explain this ACWTF???
Admin
We have a regular air conditioner and a backup air conditioner, and then the regular one for the building. The building one and the regular one fight it out sometimes in the winter (people forever are turning the heat on in the wrong place), and during one such spat the regular AC died, and the backup automatically failed over.
Unfortunately it immediately sprung a massive leak in the chiller, and started gushing water all over the place.
I walked by the door a few hours later, on my way to get my morning coffee, and my foot went squish when it hit the carpet. I had a moment of "Huh, weird, where..?" and then I no longer needed coffee.
I threw open the door on a sauna, with water bubbling from the subfloor, and many red failure LIGHTS blinking uselessly. The early guys, including one moron whose office was right next door to the damn server room, had just walked through the puddle, dum de dum dum, and not noticed a thing.
No damage, other than the dead air conditioners. All the wire was run overhead, so nothing got really wet...The ups would have been a problem, but it was in a different room. We pumped it out (fortunately we had tons of drain problems, so we had suitable pumps), fixed the AC, and I installed an old siren attached to a heat/humidity sensor.
Admin
(P.S. Server admins that can't think of anything to do but play games need to meet the same fate.)
Admin
Might be as simple as a sensor in a bad place - an old office I worked in had the thermostat immediately under a heat transfer printer. The system thought it was plenty warm and so never turned on the heat in the winter.
In your case, it could be the AC is running constantly trying to get the temperature down (and it won't because the heat source is skewing the readings).
Admin
Nobody bothered to check for Freon gas?
Admin
Do SysAdmins no longer use monitor devices? cough* nagios cough*
Admin
Then who is going to read TDWTF?
Admin
I was actually the person who submitted the original story to TDWTF (2 years ago!), and to the people who are confused about the events, I can't say I blame you, since it seems it's been nearly 100% rewritten. To clarify:
This occurred in early 2004, at the tail end of the dot-com bust, not the current recession.
The datacenter had just gone out of business. It only took 2 days for the empty NOC to trigger this condition, not weeks.
The NOC was functionally mostly for show, as NOC staff could do their jobs from either the office or the NOC. Still, during the datacenter's lifetime, there were usually at least 2 people in the NOC at all times.
No solitaire was involved, I assure you. :) Since I was the only netops member left (and one of only 3 employees left to turn down the datacenter), I was quite busy transitioning the remaining clients out for the next 2 weeks.
Ignoring the first hardware failure was true, because again, we were working on closing the datacenter, and it was a non-critical system. (Funny enough, it WAS a DNS server, but we had 4 of them, so a failure of 1 was quite low on my priority list, given the circumstances.)
Pulling the breaker on my way out was my own little literary embellishment, and is of course not true. (I was originally going to put "pushed the EPO", but I wasn't sure everyone would know what that was.)
If you'd like to read my original version, it was posted here: http://www.finnie.org/2007/07/09/if-you-cant-take-the-heat-get-out-of-the-noc/
Admin
They're not thermostats, they're on/off switches.
Admin
That would help explain how it "anticipates" all the objections of today's replies, and makes you out to be a lot more diligent and competent than the WTF version.
Admin
The other one is on the main page. up up down down left right left right b a enter
Captch: luptatum - syn. lumpy mashed potatoes.
Admin
Seems it works on most pages, actually.
Admin
I don't think he quit; I think he was the last person to leave employment when the facility was finally entirely shut down. Hence the pulling the breaker.
Admin
+1 sir
Admin
I like to describe my sysadmin job as "white-collar plumbing".
Admin
At least he kept the NOC list from getting in the open.
Admin
Burn the machine!
Admin
Our servers restart automatically when power is restored following a power failure.
The A/C in the server room does not.
Admin
had an even worse HVAC screwup in one of my buildings, but thankfully no data was involved, just people.
The new, electronically (and centrally) controlled HVAC system was wired up, but slightly off. two offices on opposite ends of a hall had their thermostats/controls swapped, so as office A got warm, the occupants would turn down the temperature. the HVAC system would respond by sending more more cool air to office B. As office B became to cold, the occupant would turn up the heat, sending more heat to office A.
it took MONTHS for someone to figure out what was happening.
Admin
"...to remind himself that things could be far worse than drawing a check to warm a chair."
Apparently they needed more people like Ryan at this company.
captcha: fakecaptcha
Admin
Love the unicorns!
Admin
People keep talking about monitoring or Nagios... and wondering why he didn't have that.
He did. It alerted him when servers went down. You'll also note in the narrative that when our protagonist asked about the temperature in the room, it read as 64 with a setpoint of 68. Because the sensor being read off was the one that controlled the HVAC unit, and the sensor was in the wrong place, the monitoring system didn't know it was 110 degrees in the server room. It did, however, alert him that things were going "poof."
A solution: have an independent temperature monitor installed. We make everything else redundant; make that redundant as well.
Admin
The other day we held a meeting and the conference room was just too small. So I got a sledge hammer and knocked down a wall to expand the conference room into the next office.
Isn't that how things are done at your company? If people see a problem, shouldn't they fix it?
Admin
Wait... then what does that make the janitors?
Admin
This should raise new questions about the cause of the WTC collapse. Maybe hot air from server rooms is what really caused the steel beams to melt?
Captcha: aptent. A new camping application installer for linux.
Admin
º is º (U+00BA), the Spanish masculine ordinal sign, which renders as a small raised letter 'o' sometimes with an underline. To get a degree sign (°, U+00B0) use °
Admin
I found myself in a job once where I literally wasn't doing anything for a year. There were three of us in that office in the same situation. One of my colleagues spent his time developing a graphics package. As for me, I busied myself writing a novel. My other colleague, indeed, could find nothing better to do than ... yep, play Minesweeper.
Admin
The real WTF is that the server room had only one breaker.
Admin
Outsourced!
Admin
Something similar happened in our office building. We had two meeting rooms, both with individual thermostats and cooling/heating. The only problem was that each of the thermostats were controlling cooling/heating in the other room.
A nice system of open feedback-loop. Guess it is a bit cold in your room, so you set the heater/cooler to heat a bit. This makes it a bit warmer in the other room, whose occupants turn on cooling. This makes it even colder in your room, prompting you to increase heating... and so on. The end result is having very cold in one room and very hot in the other despite (or rather as a result) the heater/cooler settings.
It took a year for facility management to figure out the issue.
Admin
i like the article
Admin
Since when did thermostats start thinking? The guys in our building tell us that it will take a while for the computer to respond, and if we move the lever to the up (warmer) position, it will confuse it. I should go into HVAC computer programming - sounds like some kind of broken AI.
Admin
light traction outsole and a 4 inch heel.ugg boots Women who own these shoes have been delighted and they seem perfect for those extra special occasions such as weddings. mbt shoes Sun can cause skin cancer and damages your skin. Its a known fact today that people who work out in the sun without protection from the UV rays of the sun, are 3 times more likely to develop skin cancer later on in life.