Hallway ERP

  • Tony 2007-02-27 12:03
    I find it hard to believe that the stickers sold out in less than 8 seconds...



    Note from Alex: Go figure, there was a bug in my code. I had "(hour > 12)" instead of "(hour >= 12)". It's up now, and hopefully will be for more than 15 minutes like yesterday.
  • Adam 2007-02-27 12:04
    This president was a genius! Not only did he save money by not building a new server room, but he saved money on the office heating bill by having all that high-power equipment in the hallway!

  • Ghost Ware Wizard 2007-02-27 12:08
    I can't believe the saying "if it ain't broke don't fix it" applies here but I've seen, and read, worse. Too bad the IIC <aka idiot in charge/> had to have an outside audit before he ponied up the $ for a new server room. Amazing!
  • JF 2007-02-27 12:08
    Thanks :) now time to read the article :)

    Captcha: ewww, as in the electronic world wide web? The series of tubes?
  • BradleyS 2007-02-27 12:10
    Given the way that the 100 stickers seem to be going away in the first 10 minutes of the link being up, I REALLY hope that Alex is limiting them to one per person. I want a sticker, and they're going really, really fast.
  • AbbydonKrafts 2007-02-27 12:15
    Eventually, Sarbanes-Oxley auditors came in.

    ..<snip>..

    Naturally, the auditors asked why the ERP system was set up in the hallway, and coincidentally, the next day a big fat check arrived for construction of a new server room.


    So there are good side effects for non-accounting people.. teehee..
  • finn 2007-02-27 12:16
    The sticker form works, thanks! But wasn't up exactly on 19:00 (GMT+2).

    I know a guy who was building a little server room. Kind of a semi-professional setup: good hardware, decent UPS-systems and air conditioning, etc. We laughed at him when he decided to build a DIY raised floor for no apparent reason. Well, couple of weeks from that a water pipe broke and flooded the still-not-finished server room. Luckily, the machines were alright.



  • CynicalTyler 2007-02-27 12:21
    Hey, so I needed all 100, give me a break! They're the exact right size and shape to tape together all the extension cords in our server room. Every time I trip over one we lose email for half an hour, but not any more! Thanks WTF!

    (Captcha: muhahaha? Is that an evil cow laugh?)
  • Raymond Chen 2007-02-27 12:24
    Jesus, do I need to come down there and do it myself?


    The correct response is "Yes, please. Thank you for the offer. (And stop calling me Jesus.)"
  • Tony 2007-02-27 12:26
    Ok, so how's this for a WTF... right after I posted the comment, my boss calls me into a meeting... cue 19 minutes of mindless stupidity, and I get back just in time to see more "Sorry, we're sold out today."
    Tomorrow, those suckers are mine... and if I don't get one tomorrow, I'm going to write a script to get them the next day a la the Amazon Xbox 360 fiasco LOL.

    -----------------------------------------

    Note from Alex: Go figure, there was a bug in my code. I had "(hour > 12)" instead of "(hour >= 12)". It's up now, and hopefully will be for more than 15 minutes like yesterday.
  • mdk 2007-02-27 12:52
    I don't know why, but this is the second time in two weeks that I read a story here and immediately thought of my work at the same day or the day before. (Well OK, it isn't that bad - the leak is only punctual, and a bucket can really solve the problem.)

    I think it's time to change jobs.
  • PS 2007-02-27 12:53
    Adam:
    This president was a genius! Not only did he save money by not building a new server room, but he saved money on the office heating bill by having all that high-power equipment in the hallway!



    And on really cold days the water in the now empty server room would freeze and the staff could play ice hockey. That beats a corporate gym any day.
  • snoofle 2007-02-27 13:07
    Personally, I would be more than happy to buy a sticker (how much could they possibly cost?)

    Addendum (2007-02-27 14:10):
    Hey Alex, where's the link to buy swag?


    Note from Alex: The Cafepress store remains, however I stopped linking it on the sidebar after receiving several complaints about poor customer service, quality, etc. Plus they were too expensive. Buyer beware. I will offer higher quality and lower cost swag soon from the same supplier as the Digital Donkey Mug.
  • danixdefcon5 2007-02-27 13:18
    For those that are unfamiliar, the Sarbanes-Oxley act ...


    Hm... I am in Mexico, and yet I am familiar with SOX. For some peculiar reasons, this financial institution also implements SOX. Maybe because it bought some big US branch recently?? (You must have seen it about 1 or 2 weeks ago. More specific, and I would give out where I'm working at.)

    I actually love that kind of auditors, the ones that kick stuoopid non-IT managers in the groin and make them actually spend in IT the right way.

    If only they had gone with some of my older employers ... (one of them had a "share your password with your co-workers" policy. Yeah, reeeeal secure ...
  • Jojosh_the_Pi 2007-02-27 13:21
    "Uh, yeah," replied Winston, "they're wide enough, but that's not really-"

    "Great, go ahead and start moving them."


    That exchange is the reason to start your responses with the new problem first: "This will cause a problem with X, although the width of the hallways is enough (you arrogant doofushead*)." But only add the arrogant doofushead part when it's clear your concerns are being ignored, and say it softly enough so it can't be heard over the doofushead's rambling.

    * most people would choose a stronger insult, of course.
  • Pecos Bill 2007-02-27 13:25
    Good thing I didn't work at that company. I'd be randomly switching off the servers in the hallway just for grins!
  • Marc 2007-02-27 13:32

    Stormy days, systems would be up and down.
    On clear days, the systems would still be up and down...


    :-D That's life
  • themagni 2007-02-27 13:43
    finn:
    The sticker form works, thanks! But wasn't up exactly on 19:00 (GMT+2).

    I know a guy who was building a little server room. Kind of a semi-professional setup: good hardware, decent UPS-systems and air conditioning, etc. We laughed at him when he decided to build a DIY raised floor for no apparent reason. Well, couple of weeks from that a water pipe broke and flooded the still-not-finished server room. Luckily, the machines were alright.



    See, that's one of those situations where he could never explain to you why he was raising the floor. You'd skip it if you tried to do it yourself. But no, one busted pipe and that $100 worth of materials saved you thousands of dollars in equipment and time. You'll raise the floor next time you get a server room built, won't you?

    That guy knows what he's doing. Hire him next time.

    And that's the point of this article, isn't it? You can spend just a few hundred dollars to get something fixed before it's a problem, or you're going to spend thousands of dollars to get it fixed later, once it IS a problem.

    Huh, I forgot what industry I'm talking about. ;)

    I got a sticker today. It was busted at first, but Alex fixed it PDQ. Thanks, Alex!
  • themagni 2007-02-27 13:45
    Pecos Bill:
    Good thing I didn't work at that company. I'd be randomly switching off the servers in the hallway just for grins!


    I'd switch them off and swap some cables.

    Once the boss' email went down, cheques would get written.
  • Squibby 2007-02-27 13:50
    Heh. One of the first topics that Bruce Schneier discusses in his book 'Secrets and Lies', is that CEOs will never spend money on security (and that includes availability, redundancy) until he is held accountable by his superiors, or the law.

    I see examples of Schneier being proven right every day with stories and events such as this. What's worse is that the President probably wouldn't have even approved the funding even if he were shown the Sarbanes Oxley documentation before his audit.

  • morry 2007-02-27 14:00
    I've worked at a place like that. Couldn't spend a dime on infrastructure, raises, ice machines, anything like that, but they dumped millions into the laps of consultants and other companies to implement new processes and proceedures. And they wonder why all their projects cost more $$$ these days.
  • Romanski 2007-02-27 14:03
    Uhm... I've posted something in this forum about stickers at 12:01 or thereabouts, and the post was definitely there (the first one). Was it lost by the software or was it deleted on purpose?

    I remember seeing that post about "sold out in less than 8 seconds" and mine was there too. Now mine is gone. This is not very nice :(
  • Socrates 2007-02-27 14:19
    CAPTCHA: onomatopoeia
  • Jeff S 2007-02-27 14:20
    Romanski:
    Uhm... I've posted something in this forum about stickers at 12:01 or thereabouts, and the post was definitely there (the first one). Was it lost by the software or was it deleted on purpose?

    I remember seeing that post about "sold out in less than 8 seconds" and mine was there too. Now mine is gone. This is not very nice :(


    Maybe, just maybe, if you post something actually relevant and on-topic, it won't get deleted?

    Just a crazy thought.
  • akatherder 2007-02-27 14:20
    We had our servers and a/c on a massive enterprisey UPS. The power got cut one day. As it turns out the a/c's WEREN'T on the UPS. The servers all stayed on with no a/c. These are a/c units that are about the size of 4 fridges next to eachother and we had 4 of them blowing. It got up to 140 degrees in about 15 minutes. We were shutting systems down from our desks, but that was taking too long so we just started doing manual power downs.

    Many a motherboard got smoked that day :( The place would have burned down if it happened overnight.
  • imagination 2007-02-27 14:21

    just imagine the servers being plugged into one ups that gets pulled everytime the single hallway plug is needed for the vacuum cleaner...



    captcha: atari - can anyone remember these?
  • ptomblin 2007-02-27 14:35
    I'm working at a very large company in Rochester NY that used to be a world leader in picture film. Back in September, a bunch of roof tiles in the hall collapsed because the roof was leaking. Our crack maintenance team sprung into action and suspended a plastic tarp with a hose in the middle (trade name "Leak Catcher") from the ceiling. It's now February and it's still there. Nobody has fixed the ceiling.

    I'm thinking of doing what the protagonist of this story should have done, and filed a complaint with the fire marshall about obstructions in an exit hallway.
  • Juggler 2007-02-27 14:45
    snoofle:

    Addendum (2007-02-27 14:10):
    Hey Alex, where's the link to buy swag?


    +1
  • Bill 2007-02-27 14:52
    imagination:

    just imagine the servers being plugged into one ups that gets pulled everytime the single hallway plug is needed for the vacuum cleaner...



    captcha: atari - can anyone remember these?


    Small computer story.

    Years ago, while onsite loading SW and data onto a client minicomputer, I was informed that the system kept crapping out just before 8 each morning. I checked the logs and saw that to be true, but after running diagnostics could find no issues. Power perhaps?? Nope. It's plugged into a dedicated outlet, as evidenced by the nifty red outlet and confirmed by the electrician.

    So.... I show up early the next morning and, sure enough, down it goes. So I run tests again and continue to run thru theories.

    The next morning, I'm standing there waiting for the inevitable reboot, when I see one of the draftsmen in the next room (hardly a secure server environment, BTW) come in to start his day, walk over to the blue-line (big printer for blueprints) and switch it on. Bingo!!!!

    Come to find out every elevtronic device in the building was on ONE SINGLE SHARED dedicated circuit.
  • Eric 2007-02-27 14:59
    Something similar happened a few years ago across the state lines from where I live, but not involving computers.

    There had been numerous requests for money to repair the major highway between two towns about 100 miles apart, but they couldn't get it allocated.

    So the mayors of the two towns got together and hatched a plan.

    They made up some kind of celebration and invited the governor to visit their towns. He was to land at one town for a lunch meeting and then travel with the mayors and the local state representative to the other town for a dinner meeting. The governor's airplane was to meet him in the other town that evening and fly him back to the state capitol.

    So everything was going smooth. The governor spoke at the dinner meeting to everyone who showed up and they had a nice meal.

    Then they left for the other town. The driver was a friend of my older brother.

    As they were going down the bumpy road, with the governor feeling the bumps, they reached a point where the road was too bad to drive on. So they pulled into the ditch for a quarter mile or so and then back onto the highway. While they did this, the two mayors kept their discussion going like nothing unusual was happening. They didn't even look around at the highway or the ditch.

    But the governor sure did notice. The driver said that he almost snapped his neck looking over toward the road and that from then on he paid more attention to the condition of the major highway than he did the conversation.

    But noone in the car made the slightest comment about the road.

    That night, the governor returned to the state capitol, and the next day, he found enough funds for the highway department to repair that highway.
  • Moss 2007-02-27 15:09
    Why do people seem to think computers are so great that they can't overheat or die?
  • cparker 2007-02-27 15:16
    ptomblin:
    I'm thinking of doing what the protagonist of this story should have done, and filed a complaint with the fire marshall about obstructions in an exit hallway.
    You may have just done exactly that, o dilligent worker of Eastma^H^H^H^H^H^Han anonymous photo film company. ;)
  • kimbo305 2007-02-27 15:26
    Moss:
    Why do people seem to think computers are so great that they can't overheat or die?

    well, there's certainly no warnings on most boxes saying, hey, ventilate me, or diagrams of good infrastructures for your data center. The commoditization of hardware has given people the impression that servers are just plug and play, and don't need to be farmed carefully.
  • Rob Sirloin 2007-02-27 15:26
    WOW. That really IS "Worse than Failure"....

    Nope, still not working for me. This is gonna take some time to get used to.
  • iAmNotACantalope 2007-02-27 15:29
    snoofle:
    Personally, I would be more than happy to buy a sticker (how much could they possibly cost?)

    Addendum (2007-02-27 14:10):
    Hey Alex, where's the link to buy swag?


    Meanwhile, 5 days * 100 stickers * $0.39 = $195. That's a lot to mail some stickers. Plus Alex's time to stuff envelopes.

    And I wouldn't be surprised to see the envelopes hand addressed and stamped, just for the WTF factor. (And don't manually correct the &amp !)
  • Ohnonymous 2007-02-27 15:43
    The REAL WTF is that they were solving Rubik's cubes in the 80s. No wonder they couldn't get funding when they really needed it.
  • Dazed 2007-02-27 15:46
    themagni:
    Pecos Bill:
    Good thing I didn't work at that company. I'd be randomly switching off the servers in the hallway just for grins!


    I'd switch them off and swap some cables.

    Once the boss' email went down, cheques would get written...


    ... for security cameras, to see who was swapping cables.
  • iAmNotACantalope 2007-02-27 15:48
    Dazed:
    themagni:
    Pecos Bill:
    Good thing I didn't work at that company. I'd be randomly switching off the servers in the hallway just for grins!


    I'd switch them off and swap some cables.

    Once the boss' email went down, cheques would get written...


    ... for security cameras, to see who was swapping cables.


    You mean the RJ45 doesn't go in the floppy drive? I mean it rhymes... apparently.
  • bob the dingo 2007-02-27 15:50
    I was the IT manager at my last job, and had to go through an S-O audit. It really is good for getting some extra money for stuff like that despite management's best efforts!
  • Pitabred 2007-02-27 15:55
    danixdefcon5:
    If only they had gone with some of my older employers ... (one of them had a "share your password with your co-workers" policy. Yeah, reeeeal secure ...


    Before I was paying attention (I'm a solutions engineer, and I just reset my own password as I see fit, I don't deal with IT that much, except to request upgrades), apparently our old IT guy had all the user passwords written down on a sheet of paper on his desk. And none of our other users questioned it :( He never asked me for mine. At least we have someone competent in now... I was joking with him about how if you have root access, you don't need no steenkeng user passwords.
  • Pitabred 2007-02-27 15:57
    No, the new acronym isn't really working... but it's more PC. I figure that if you really know what WTF means, you don't need to worry about it, and now you can explain what you're doing to your boss ;) Heading off problems at the pass, before a project does something worse than failing... yeah, that's it...
  • Sammy 2007-02-27 15:59
    ptomblin:
    I'm working at a very large company in Rochester NY that used to be a world leader in picture film. Back in September, a bunch of roof tiles in the hall collapsed because the roof was leaking.


    Jesus. Knowning what I know about where you work, it's not exactly like you guys don't get much rain, either.

    Along similar lines: we once had people working on the roof over our server room who left the tiles completely off overnight. None of the roofers, of course, had bothered checking the weather forecast, so when I came in the next morning under a slow, steady shower, I discovered my coworkers frantically throwing tarps over everything in our ops room. Water was streaming in from various points in the ceiling.

    When the boss bawled out our landlord, the landlord responded, somewhat defensively, that his roofers "didn't think it was going to rain." Well, I suppose that was obvious.
  • Muffhead 2007-02-27 16:01
    I was in the computer room for a small local bank about 10 am on a Friday morning. On the last day the last day of the month. I started to notice drips. My coworker then noticed drips on the other side of the room. The ceiling mounted AC was right in the middle of the room & spewing out water. Called the AC company, called the AS/400 guys & got plastic bags & buckets. The AS/400 guys were on standby to relocate the system out into the hall if needed. By the time the AC guys showed up there was a nice regular drip & it was almost lunch time. Told the floor manager to prepare to have the tellers run with paper only if needed. The look on her face was priceless, but she was prepared.

    Finally with us covering the AS/400 & racks with plastic sheets & holding buckets for the AC guys they fixed the unit. We never had to take them down & the client was very happy.

    Went back to the office & got my head ripped off by the boss for wasting my time, coworker's time, AS/400 guy's time & worrying the client unecessarily.

    Thankfully I'm not with that company any more. The bank never moved the AC.
  • MX5Ringer 2007-02-27 16:05
    Ok guys,

    after the slating I got earlier for being 'Frist' (which has now been removed!)

    I won't waste anybodys time by getting my own back for the truly hurtfull comments made by some!

    This WTF actually rings so true with me.

    On more than one occasion I'v had to take off my shoes and roll my trousers up to wade into my server room during heavy rainstorms (Manchester UK) as the roof leaks so much (the general office staff have much the same problem, more of a wet carpet problem for them though rather than a pissing in through the ceiling problem that I have)

    The directors think the solution is to have the power and network points raised up the walls and the server racks repositioned above the max waterline rather than have the roof fixed!!

    Thanks,
    That is all.

    (Actually if I submitted half the things that happen to me daily every day you wouldn't believe it. That's why I'm looking for a new job!)

    CAPTCHA: Pointer, where's mine gone? can't see it under all this water!!
  • anony-mouse 2007-02-27 16:13
    Boo, I wish we had some more real stories instead of this fiction.
  • ptomblin 2007-02-27 16:15
    cparker:
    You may have just done exactly that, o dilligent worker of Eastma^H^H^H^H^H^Han anonymous photo film company.


    Well, in that case I should mention that is is building 205 of that anonymous photo film company, somewhere near the Entertainment Imaging offices.
  • Daniel Vandersluis 2007-02-27 16:18
    themagni:
    Pecos Bill:
    Good thing I didn't work at that company. I'd be randomly switching off the servers in the hallway just for grins!


    I'd switch them off and swap some cables.

    Once the boss' email went down, cheques would get written.


    ... for security cameras to watch the hall.
  • Daniel Vandersluis 2007-02-27 16:23
    Ha. I was beaten to it, and didn't even notice. Duh.
  • Nefarious Wheel 2007-02-27 16:36
    After a long day finishing up at Denny's at 1am on new year's day, the support phone went off -- me and the support manager were enjoying a cleansing ale or two and had to support a client in Florida on the other side of the country, over the phone.

    After listening to Mike speak for about half an hour, things like "Turn the console key to the horizontal position -- you know, if you were lying down, that's horizontal, turn the key that way " to an operator who spoke fluent Spanglish in a near-supersonic tone of voice, he escalated the call. "It's for you" he said, and handed me the phone.

    After about an hour of diagnosis (as in register pokes and peeks -- it was an old General Automation mini, a "JumboGA") I was convinced it was a hardware failure, and handed it back to Mike.

    Mike patiently explained that their service contract was current, an engineer would be on a plane first thing in the morning, it was all paid for etc... but his offers for help were rejected.

    "You can't get an engineer here, the airports are all closed." "Why?" asked Mike. The customer replied "Because there's a hurricaine in the Gulf of Mexico and the airplanes can't land. In fact there's six inches of water in the computer room right now".

    --==[You know who you are]==--
  • Patrick 2007-02-27 17:09
    So my friend the artist said to me, "How's work going?"

    And I told her, "Say your boss asked you to make a painting...but you weren't allowed to use a brush."

    She goes, "I'd fingerpaint, or just dump the paint on the canvas and call it abstract art."


    And I go, "Well, I'm fingerpainting."
  • JTK 2007-02-27 17:20
    Patrick:
    So my friend the artist said to me, "How's work going?"

    And I told her, "Say your boss asked you to make a painting...but you weren't allowed to use a brush."

    She goes, "I'd fingerpaint, or just dump the paint on the canvas and call it abstract art."


    And I go, "Well, I'm fingerpainting."


    Ah, a VB programmer....
  • Nathan 2007-02-27 17:21
    Bill:

    Small computer story.

    Years ago, while onsite loading SW and data onto a client minicomputer, I was informed that the system kept crapping out just before 8 each morning. I checked the logs and saw that to be true, but after running diagnostics could find no issues. Power perhaps?? Nope. It's plugged into a dedicated outlet, as evidenced by the nifty red outlet and confirmed by the electrician.
    Come to find out every elevtronic device in the building was on ONE SINGLE SHARED dedicated circuit.


    "Shared dedicated" is one of my favorite marketing terms. You sell someone on the fact that they own their slice of the machine, forget the fact that no matter how well you virtualize, someone can take down the whole thing.

    Captcha: Darwin - his ghost walks the halls where I work, waiting for ideas like shared dedicated to ensure the survival of the fittest.
  • Jeff 2007-02-27 17:36
    Are you part of the cleaning crew at my office? Every few months I have to have conversations with them for this problem. Because, of course, it's too expensive to run 10' of wire and put in a new breaker for the server UPS's. and you always wait until I am at least 100 miles out of town before you decide to use that plug....WTF
  • Jeff 2007-02-27 17:39
    imagination:

    just imagine the servers being plugged into one ups that gets pulled everytime the single hallway plug is needed for the vacuum cleaner...



    captcha: atari - can anyone remember these?


    Are you part of the cleaning crew at my office? Every few months I have to have conversations with them for this problem. Because, of course, it's too expensive to run 10' of wire and put in a new breaker for the server UPS's. and you always wait until I am at least 100 miles out of town before you decide to use that plug....WTF
  • dkf 2007-02-27 17:45
    akatherder:
    We had our servers and a/c on a massive enterprisey UPS. The power got cut one day. As it turns out the a/c's WEREN'T on the UPS. The servers all stayed on with no a/c.

    Been there. We had the additional "fun" of finding out that the emergency lighting in the whole building (including the windowless machine room) also wasn't on the UPS and there were a number of gaping holes in the floor in the machine room at the time. The net effect reminded me of Doom 3 (though without the demons, unless you count the operators.)
  • Andrew 2007-02-27 17:59
    The last of the IT budget had just been spent on free stickers for TDWTF's readers...
  • TheRealBill 2007-02-27 18:38
    themagni:

    I got a sticker today. It was busted at first, but Alex fixed it PDQ. Thanks, Alex!


    Dear WTFer. It has come to our attention that the D in PDQ stands for Damned, which is offensive to some readers as a curse word. in order to align with the rewording of the site name to worsethanfailure to avoid the connotations with the F-word often associated with WTF, we need you to stop using PDQ and instead type out prettydarnedquick in order to not offend those who may be offended.

    At least until grandma starts pointing out that darned is also a knitting term and thus not appropriate to use because some knitters may object to the use of darned in a potentially negative connotation. At which point we will fallback to our second choice which is plentydonequietly. Yes we understand that it has no bearing on what you intended, but grandma will indeed be pleased.

    Sincerely,
    The New Management

    ...
    hehe
  • TheRealBill 2007-02-27 18:39
    Romanski:
    Uhm... I've posted something in this forum about stickers at 12:01 or thereabouts, and the post was definitely there (the first one). Was it lost by the software or was it deleted on purpose?

    I remember seeing that post about "sold out in less than 8 seconds" and mine was there too. Now mine is gone. This is not very nice :(


    Dude, that's worse than failure.
  • TSR 2007-02-27 19:32
    I worked for a power station that had set a new record price when it was privatised and had been losing money steadily ever since. All our comms data and phone were located in a large building once filled with (power) network testing staff but was now empty except for our gear and rooms full of boxes of paper archives. After a heavy downpour we found the ceiling tiles over the PABX had collapsed from the weight of water and the console had shorted. The was also ominous drips over the data comms racks.

    The solution... beautifully made sheet metal 'umbrellas' were suspended over all critical equipment with the idea that any water would run done into the deserted ground floor (umm, what about those archives guys?).

    Eventually repairmen came and stipped off the old bitumen roof coating to repair it and then decided the weather didn't look so good and left for about a year. We ended up with sandbagged tarps on the roof until the following spring. They never really understood that no a/c and lots of dust and moisture were a bad environment. After all the coal dredgers worked in it!

    captcha: bathe
  • simon 2007-02-27 19:37
    The best one of these I've seen was the data center at a research facility for one of the biggest car companies in the world. Huge place 100's of acres, amazing offices, with cars INSIDE, to be talking points in various reception areas etc. (they must have had a giant elevator somewhere and some big ass dollies).
    Anyhow, I was there to install some new routers, escorted down to the basement, then to the sub-basement - bare concrete walls, bare light bulbs, planks setup on milk crates to walk on because of the 6 inches of water everywhere! No IT kit anywhere in sight, finally we reach the datacenter, nice secure door, then inside raised flooring, dropped ceiling, looks like a proper server room.
    I start installing kit and lift a tile to find some power - only to find 6 inches of water and lots of leaves and mud. The cables were carefully suspended from the top of the false floor support posts, out of the water, but once good rain could change that...
  • Josh 2007-02-27 20:16
    dkf:
    We had the additional "fun" of finding out that the emergency lighting in the whole building (including the windowless machine room) also wasn't on the UPS


    Erm, most of the emergency lights I've seen have their own battery, and only turn on when the power to them is cut. Plugging those into a UPS kind of defeats the purpose.

    CAPTCHA: poindexter
  • danixdefcon5 2007-02-27 21:56
    Josh:
    dkf:
    We had the additional "fun" of finding out that the emergency lighting in the whole building (including the windowless machine room) also wasn't on the UPS


    Erm, most of the emergency lights I've seen have their own battery, and only turn on when the power to them is cut. Plugging those into a UPS kind of defeats the purpose.

    CAPTCHA: poindexter


    ...except they may have taken my residential complex's "logic".

    All the emergency lights are actually powered not by a battery, but by the emergency plant.

    And guess why I know this: the circuits were overloaded (each phase supplied current to around 20 apartments, and to my dismay, one of those phases supplied my tower's 20 apartments and the 4 elevators for the whole complex (one elevator per tower, though).

    Obviously, my tower had frequent blackouts due to the fuses popping. The owner wouldn't listen to us about needing to balance/separate into more phases. He kept only buying bigger and bigger fuseboxes (last one was a big-ass 200-amp breaker); the actual problem to solve is that the complex still doesn't have a substation to distribute power to our apartments.

    Finally, and much like this WTF ... one of the watthometers exploded. Yes, exploded. It literally went down in flames, with all the neighbors trying to put it out when I came to the apartment. Geeze, it even looked like something out of the BOFH stories. (Good thing all my computing equipment was offline).

    3 days later, the administration resigned, and the new condo admin rebalanced the power lines. Still waiting that substation, though...

    So I wouldn't be surprised by battery-less "emergency" lights...
  • brandon 2007-02-27 22:04
    Oh this is so precious!
  • Anon 2007-02-27 22:39
    Damn, I just realised - I see this every day and have gotten used to it.

    Several months ago, dripping water from an air conditioner into a power supply created pretty sparks, and stuff didn't work so well. So I built an "umbrella" out of a cardboard box and a long plastic stick. It's still there, and will probably be there for years to come.

    I feel sad now.
  • OMFG, a foreigner 2007-02-27 23:32
    "Sorry, all out for today. Try tomorrow at noon."

    Which noon? We don't all live on the same time zone.
  • felix 2007-02-28 00:17
    Jeff S:
    Romanski:
    Uhm... I've posted something in this forum about stickers at 12:01 or thereabouts, and the post was definitely there (the first one). Was it lost by the software or was it deleted on purpose?

    I remember seeing that post about "sold out in less than 8 seconds" and mine was there too. Now mine is gone. This is not very nice :(


    Maybe, just maybe, if you post something actually relevant and on-topic, it won't get deleted?

    Just a crazy thought.


    I did exactly that a while ago. I made a comment about not being first AND something on-topic. Though the forum is bristling with posts like that, mine was deleted.

    I guess our moderators act somewhat randomly...
  • akatherder 2007-02-28 01:01
    OMFG, a foreigner:
    "Sorry, all out for today. Try tomorrow at noon."

    Which noon? We don't all live on the same time zone.


    It's EST. I live on the east coast of the US and it popped up around my noon today.
  • Remco G 2007-02-28 02:49
    felix:
    I did exactly that a while ago. I made a comment about not being first AND something on-topic. Though the forum is bristling with posts like that, mine was deleted.

    I guess our moderators act somewhat randomly...

    I think that by now everyone is so fed up with spamming kids trying to get first post, I'd very much welcome it if every post that just mentioned being first got deleted.
  • The DailyBTW 2007-02-28 03:41
    Remco G:
    felix:
    I did exactly that a while ago. I made a comment about not being first AND something on-topic. Though the forum is bristling with posts like that, mine was deleted.

    I guess our moderators act somewhat randomly...

    I think that by now everyone is so fed up with spamming kids trying to get first post, I'd very much welcome it if every post that just mentioned being first got deleted.


    This is worse than funny! The politically correct way, of course, would be to always delete the first post. That way, there would be no first post and no one would be offended.
  • Ebbe 2007-02-28 03:54
    There's another WTF buried here: Winston and his colleagues did not speak the right language with management. Saying that there is a technical problem will only get a positive result in very rare cases.

    What they should have done, was to tell management that there is a potential problem that will cost the company $AAA if it materializes and it will cost $BBB to correct it - with AAA being much larger than BBB. AAA may be inflated slightly and BBB must be inflated sufficiently such that you still have emough money for solving the problem after management has cut the amount down.

    And always, ALWAYS, remember: the ONLY language management knows is money.
  • Watson 2007-02-28 04:05
    Moss:
    Why do people seem to think computers are so great that they can't overheat or die?

    You have to remember that many people don't notice the ventilation slots, and think that the fans are what makes the computer go - you know, like aircraft propellors.

    Captcha: Poindexter - Why do I have to get the difficult ones?
  • cardboard box 2007-02-28 04:48
    Thats exactly it. You have to talk managemnt the spech of money. If this is not fixed now it willl cost toe company XXX$ per minute if this system fails, etc... :D
  • dkf 2007-02-28 04:57
    Josh:
    Erm, most of the emergency lights I've seen have their own battery, and only turn on when the power to them is cut. Plugging those into a UPS kind of defeats the purpose.

    Congratulations on spotting one of the WTFs in that situation. Another (OK, not explicitly stated) was the fact that none of the routers had any backup power at all, so lots of critical software keeled over anyway despite being on machines with a proper power strategy. To cap it all, I think the internal phone network also failed; good thing I had my mobile with me...

    We've not had a full power failure since then so who knows whether the situation was fixed. But the H&S people were very upset about the failure of the emergency lights. Thankfully I work in a different part of the building now, with windows so there's at least some external light.
  • General Dogsbody 2007-02-28 05:55
    I worked for a pensions company in the UK before the my current job and we had a business centre some 200 mules foun our HQ that suffered a network crash in their router every afternoon at 4:30pm which would require a cold boot of the aforementioned router.

    At the time I worked on second line support and was tasked with working out what was happening. We knew that the cleaners arrived around then and that the cupboard housing the router also had their vacuum cleaner but the obvious "unplugging" was not happening.

    Eventually we worked out that the router was bolted to a shelf behind the door (all cables and buttons facing the wall) and that the cleaner was forcing the door back to get the vacuum cleaner out and hitting the router. This would cause it to slide on its shelf and push it against the wall causing the 3.5" floppy disk to partialy eject. When the router was rebooted what drive would suck the disk back in and everyhting would be alright until the next night.

    Knowing that we were scrapping that router in a few weeks time and not having any spanners to hand I got the office manager to snap the eject button off as a temporary fix!
  • Alex 2007-02-28 07:01
    Remco G:
    felix:
    I did exactly that a while ago. I made a comment about not being first AND something on-topic. Though the forum is bristling with posts like that, mine was deleted.

    I guess our moderators act somewhat randomly...

    I think that by now everyone is so fed up with spamming kids trying to get first post, I'd very much welcome it if every post that just mentioned being first got deleted.

    Along with every post where users post their fucking captchas!
  • GJ 2007-02-28 07:22
    Classic stuff! Sounds just about right. "IT costs way too much and we never get anything back!" OK, well next time you want to fulfill all those back orders on a Friday night and Word/Excel ain't working and the DB's ain't available you'll find me in the local boozer, toasting your ability to save another few pence!


    I give up trying to get a sticker, so stuff it, I'm gonna buy a pack of labels at Office World and print my own on the company colour printer...and they'll have hookers and blackjack! Forget the stickers!
  • RAMBOTRAN 2007-02-28 07:46
    Been there, got that tee-shirt.

    Worked at one of those global oil companies in a UK computer centre for a while. They spent six months installing a giant Caterpillar genset and literally lorry loads of batteries to keep some IBM 3090s and a pile of VAXen running.

    The day of the test came and the senior managers came to watch the mains power being cut...and lo! the Caterpillar did burst in to life and take the load from the batteries and much back-slapping began.

    Five minutes later the high temperature alarm inside the computer room activated and cut all power. Have never heard a silence as silent as that ever.
  • akatherder 2007-02-28 09:06
    It seems like most pointy hairs sit on one extreme or the other. One group will spend billions making sure every thing is redundant times ten. They'll throw money at anything they think helps cover their ass. Let's be honest though, if the office burns to a crisp, there's going to be an interruption in service.

    Then there's the more common group who will ask what the cost is. Then they ask what the cost to recover is. Then they ask what the chances of a catastrophic shutdown are. If you don't quote 100% chance of failure, they are gambling with company money! You'll be lucky to get a second power cord for the dual power supply servers out of them.
  • poochner 2007-02-28 09:10
    Back in my college days twenty-odd years ago, I worked for the campus data center. We used mostly CDC mainframes, with a couple of IBMs thrown in. All the Vaxen and whatnot belonged to the compsci people, and were in their own center. One February, some semi-literate genius from facilities notices that the chillers in the building next door are running, so he shuts them off, it being cold out and all that.

    You should have heard all the alarms going off in our building when the water-cooled boxes lost pressure and shut themselves down. There was much running about and waving of hands. Since the only major job running at the time that got hosed was campus payroll, no disciplinary action was taken -- other than letting people know the details of why payroll was held up. After that, only a couple of people had keys to the chiller building.
  • Romanski 2007-02-28 09:56
    Jeff S:
    Maybe, just maybe, if you post something actually relevant and on-topic, it won't get deleted?

    Just a crazy thought.


    Crazy, indeed: lots of other posts about stickers exist which were not deleted. So I doubt it was censorship. Software failure? Easily. Should it be fixed? Yes, please! :)
  • Rich 2007-02-28 09:58
    AbbydonKrafts:

    So there are good side effects for non-accounting people.. teehee..


    Sarbanes Oxley got us the proper backups and redundant servers that we'd been saying we needed for two years. Execs go into meeting all smiles, come out kanda pale (parts of SOX are enforced by jail time) and writing checks...

    Rich

  • Laurence F. Sheldon, Jr. 2007-02-28 10:49
    A/C failures were common in the Data Center in San Jose because it sometimes gets beastly hot, we had rows and rows of discs (85 F exhaust air) and huge CPU's (exhaust from an 1110's CPU power supply stack was enough to cook lunch in), and the cooling tower was a bad compromise located below ground. And the tower was old and chronically scaled up.

    So the PHB (term not invented yet, but the prototype I believe was a peer of the one Adams used--same company) suggested we stock a collection of window-mounted air conditioners to be deployed on chairs around the Data Center when the need arose.
  • Wene Gerchinko 2007-02-28 11:15
    Ebbe:
    [snip]
    ...And always, ALWAYS, remember: the ONLY language management knows is money.


    I agree it's always better to speak to Mgt with that in mind. I call that move "quantifuscation".

    Managers also love new buzz words that make NO sense.

    captcha = dubya (how appropriate)
  • Autocracy 2007-02-28 11:36
    This is the second reason for using the BigRedButton
  • Laurence F. Sheldon, Jr. 2007-02-28 12:06
    Seems line 1202 would have gotten me in for a sticker.

    Are they real? And how appropriate--the captcha is "gotcha".
  • Paul Nieuwkamp 2007-02-28 16:17
    Sounds familiar. Here in the Netherlands we have the "ARBO-dienst". It is some agency that checks all kinds of rules about safety on the workplace etc.

    One day I went to my parents job. About the first thing I noticed when I got into the warehouse were the improvements on the fork-lift. The lights were working again, it had a seatbelt etc -> The "ARBO-dienst" had been there to check everything.

    The broken light, non-existent seatbelt and the other issues had been there for a few years. Not that it made working any less safe or anything, but the "ARBO" requires working lights, a seatbelt etc.

    The reason nobody ever bothered to fix this was not even management/money-related, the fixes were (or, are) useless. The forklift is used, 6 times a month, 10 seconds per 5 minutes for about two hours, inside a well-lit loading dock.
    It literally goes up 4 feet and moves forward one feet, to be reversed 5 minutes later and repeated 5 after that.

    If I recall correctly that was the only issue they found, so that was pretty good actually.

    Totally unrelated: "Erp" is the name of the village I live in. The first time I heard about ERP-software I was quite confused :P

  • TSR 2007-02-28 23:08
    Back to the power station again (I saw a few WTFs there). We used to boast that we had the biggest backup generator, 1490MW of coal fired turbines, until the day a contractor was doing some work in the switchyard and decided to carefully lockout the main feed to the control room... and the backup feed.

    In the control rooms all the lights and panels went dark so the operators hit the emergency trips on the turbines (without control, turbines race until they shed a few blades. You do *not* want to be in the same building when that happens).

    When the lights went out in the office I walked out the door and heard for the first time the sound of a power station winding down. It took 3 days to get back up and lost us around $20M.

    Oddly our main UPS in the server room didn't kick in. After the power came back on the Network Manager reset the unit and it blew a capacitor very loudly. That woke us up.
  • cklam 2007-03-01 14:12
    Dazed:
    themagni:
    Pecos Bill:
    Good thing I didn't work at that company. I'd be randomly switching off the servers in the hallway just for grins!


    I'd switch them off and swap some cables.

    Once the boss' email went down, cheques would get written...


    ... for security cameras, to see who was swapping cables.


    You must be joking ... in that kind of place ? That sounds like the place where they would call the IT people for anything electrical ... you would know about cams being installed.
  • slowtiger 2007-03-05 06:55
    imagination:

    just imagine the servers being plugged into one ups that gets pulled everytime the single hallway plug is needed for the vacuum cleaner...


    Actually this happened at the company I spent some life with. The server room was going to get air condition finally. Some serious reconstruction took place. During this process, on power line came out from that room (it was still small enough to be fed by just one chord) and was plugged in an ordinary wall socket in our graphics department. A sign placed on top of that indicated severe treatment of any puller.

    One morning I came in to find hell bent loose. Customer servers had been down as well as other machines. Somebody had pulled the plug, and it hadn't been the janitor. When they followed the line, they found a cell phone recharger plugged in instead. Complete with the phone. It was not so hard to find out which manager's phone it was.