• (cs)

    Opps. Oh, I mean UPS.

  • Someone (unregistered)

    Kind of silly if you think about it. A UPS powering a whole farm of servers (which are designed for two) and another UPS? Seems a bit of a ridiculous mistake to make.

  • My Name Is Missing (unregistered)

    The requisition was filed next to the Ark Of The Covenant.

  • (cs)

    The 9V batteries in the picture appear to be wired in series, which does not provide redundancy. Although, if surreptitiously applied to a computer mouse, they would provide hilarity.

  • foo (unregistered)

    I see the problem: Let's say the servers are in Rack A and Rack B. UPS 1 was providing backup power to Rack A. UPS 2 was providing backup power to Rack B, and primary power to Rack A UPS 1 was plugged into UPS 2, but UPS 2 couldn't provide enough power.

    They needed to plug the primary power for rack B into UPS 1, and UPS 1 into UPS 2. Then they'd have had all the servers on two UPSs, and each UPS on a UPS. As a bonus, if the power failed, both UPSs would still have been charging while they were powering the servers, so their batteries would not have run flat.

  • anonymous_coder() (unregistered) in reply to frits

    Yep - ever welded with 9V batteries in series? You can scare the hell out of someone doing it :)

  • socknet (unregistered)

    I didn't really understand this, but I want to comment anyway.

    Power to the people. TRWTF is UPS.

  • Jorf (unregistered)

    Yo dawg I herd you like UPS so I plugged a UPS in your UPS so now you can power your servers while you're powering your servers.

  • (cs)

    I blame Hubert. He checked the UPSs after the first crash. He should have seen how everything was plugged in and fixed it then.

  • Zapp Brannigan (unregistered)

    Whoa, whoa, just be careful you don't feed the supplies into each other because you might create some kind of perpetual motion machine. Our corporate policy is to obey all the laws of thermodynamics.

  • Meep (unregistered) in reply to DeaDPooL
    DeaDPooL:
    I blame Hubert. He checked the UPSs after the first crash. He should have seen how everything was plugged in and fixed it then.

    It was too dark.

  • veggen (unregistered)

    I love how the article is written. Points for Remy.

  • Gary (unregistered)

    Message left on server: "UPS Delivery Problem NR 27743"

  • Machtyn (unregistered) in reply to Jorf
    Jorf:
    Yo dawg I herd you like UPS so I plugged a UPS in your UPS so now you can power your servers while you're powering your servers.

    Sadly, this is extremely relevant to the topic at hand.

  • Ixitar (unregistered) in reply to DeaDPooL
    DeaDPooL:
    I blame Hubert. He checked the UPSs after the first crash. He should have seen how everything was plugged in and fixed it then.

    I agree. Why is Hubert involved? Where are the people that setup the UPSs?

    He is either responsible for the server room or he is not. If he is, then he should be making an issue of the problem and fixing what he can. If he is not, then he should stay away from the server room.

    Where is the source of this story?

  • Fritz Lang (unregistered)
    [image] Who hasn't done something like this?
  • (cs)
    Since an inning in baseball is roughly equivalent to an over in cricket

    I know nothing about baseball, but I'm pretty sure you're wrong there... (http://en.wikipedia.org/wiki/Innings)

  • booger (unregistered) in reply to Fritz Lang
    Fritz Lang:
    [image] Who hasn't done something like this?
    If someone did something like this at my job, I'd use it to weld their nuts to their underwear.
  • rfoxmich (unregistered) in reply to Ixitar

    I blame the system.

  • Top Man (unregistered)
  • (cs)

    TRWTF is top men. If you want this handled, get a top woman.

  • n9ds (unregistered)

    Where I used to work the LAN Admin set up a new server which had 3 redundant power supplies and plugged all three into the same surge suppressor. Oh and the surge suppressor had a reset/trip, so that when the power went out, you had to manually reset it. At least they plugged that into a UPS.

  • (cs) in reply to veggen
    veggen:
    I love how the article is written. Points for Remy.
    Please sterilize yourself. I found this article utterly excruciating to read. Remy reminds me of this guy at the office who loves to use obscure words at the expense of clarity, and usually uses them wrong anyway.
  • (cs) in reply to Top Man
    Top Man:
    You aren't that TOPS programming language guy are you?
  • riiiiiiight (unregistered) in reply to Ixitar
    Ixitar:
    DeaDPooL:
    I blame Hubert. He checked the UPSs after the first crash. He should have seen how everything was plugged in and fixed it then.

    I agree. Why is Hubert involved? Where are the people that setup the UPSs?

    He is either responsible for the server room or he is not. If he is, then he should be making an issue of the problem and fixing what he can. If he is not, then he should stay away from the server room.

    Where is the source of this story?

    Yeah, because we've never heard of anyone being in a situation where his responsibility outstripped his authority.

  • Just Me (unregistered)

    For those too lazy to view the source: click on "Poor little UPS #2".

  • (cs) in reply to foo
    foo:
    I see the problem: Let's say the servers are in Rack A and Rack B. UPS 1 was providing backup power to Rack A. UPS 2 was providing backup power to Rack B, and primary power to Rack A UPS 1 was plugged into UPS 2, but UPS 2 couldn't provide enough power.

    They needed to plug the primary power for rack B into UPS 1, and UPS 1 into UPS 2. Then they'd have had all the servers on two UPSs, and each UPS on a UPS. As a bonus, if the power failed, both UPSs would still have been charging while they were powering the servers, so their batteries would not have run flat.

    Troll science, anyone?

  • (cs)

    Heh. Ours is much nicer, as befits a company that has the sort of servers that MUST be shut down in a controlled manner. Every wall outlet in the server room runs to a huge battery backup that can handle everything for about 15 hours.

    We had an outage a few years back, knocked out the whole building. I got up when my pager started making frantic sounds, checked the power, went back to bed, got up an hour later, checked it to make sure it wasn't nosediving...No worries. Plenty of time.

    At about 6:00am, my pager screamed again. Different tone. It's the 25% power warning. What the hell? I pulled on some clothes, and laid a patch pulling out of the driveway. Halfway to the building, pager screams again. THERMAL WARNING. One of the batteries must have caught fire or something! I slammed the pedal to the floor, and tried to keep from hyperventilating.

    I got to the building, jumped out of the car, sprinted for the door. Took the stairs a landing at a time, and kicked into the server room to find...That a moron coworker had allowed some customer service reps to set up shop in the server room.

    (Pause for people to say...WHAT?)

    Yes. I suppose the logic went as follows: "Well, they need to do their jobs, and theres only one place in the building with any power, sooooo." The practical upshot was that he'd lugged a few desktops in, and plugged in a few phones, and then left them there and gone back to bed. At which point they decided it was too cold, and went and got their space heaters.

    I suppose they were just going to sit and work until the financials mainframe caught fire and/or ran out of power and performed an uncontrolled shutdown on itself. I completely lost my shit and threw them out, raving and twitching like a pentacostal street preacher. They complained to their boss, who did his best to get me fired because a) I yelled at them, and b) we had to pay for more time at our remote call center in India and that's expensive.

    I protested on the grounds of all the rules vis a vis our server room security, meant to cover machines that generally dealt with 6 figures a day in cash transactions, and the health and well-being of our servers. My concerns were laughed off at a local level, but I'd bbc'd one of the regional admins, who took it to the CTO who had an exorcist-style screaming fit, which resulted in a lot of people (not including me) getting fired.

    I learned my lesson about battery backups. When you get the call, even if you have hours left, you've got to supervise them. It's like a death watch. Also learned a lesson about the kind of person who has server room access for the purposes of changing tapes and stuff like that.

  • (cs) in reply to Zylon
    Zylon:
    veggen:
    I love how the article is written. Points for Remy.
    Please sterilize yourself. I found this article utterly excruciating to read. Remy reminds me of this guy at the office who loves to use obscure words at the expense of clarity, and usually uses them wrong anyway.

    *wrongly

    Evidently some people can't use simple words correctly either... :P

  • Robo (unregistered) in reply to Zylon
    Zylon:
    veggen:
    I love how the article is written. Points for Remy.
    Please sterilize yourself. I found this article utterly excruciating to read. Remy reminds me of this guy at the office who loves to use obscure words at the expense of clarity, and usually uses them wrong anyway.
    Agreed. My first question was why the UPS alarms had been going off for a year before Hubert decided to do something about it? Had to re-read before it made any sense
  • socknet (unregistered) in reply to DaveyDaveDave
    DaveyDaveDave:
    Since an inning in baseball is roughly equivalent to an over in cricket

    I know nothing about baseball, but I'm pretty sure you're wrong there... (http://en.wikipedia.org/wiki/Innings)

    Absolutely no idea where you are quoting from.. but it is much more interesting that UPS, so I'll go with it..

    an Innings in baseball is roughly equivalent to... shock.. an innings in cricket. I realise this may suprise a lot of people, so I'll give you some time to recover before making any further major revelations.

  • (cs) in reply to Satanicpuppy
    Satanicpuppy:
    I got to the building, jumped out of the car, sprinted for the door. Took the stairs a landing at a time, and kicked into the server room to find...That a moron coworker had allowed some customer service reps to set up shop in the server room.

    (Pause for people to say...WHAT?)

    I call shenanigans on this story. I've worked in many different call centers, and I'm aware of the consistent level of professionalism shown by management at every company.

    There is no way you wouldn't have gotten fired for kicking the CSRs and their space heaters out of the server room. :)

  • (cs) in reply to Just Me
    Just Me:
    For those too lazy to view the source: click on "Poor little UPS #2".

    WTF! That is too funny!

    [image]
  • (cs) in reply to socknet
    socknet:
    Absolutely no idea where you are quoting from..

    Have a look at the source - it's where the best bits of Remy's articles are to be found :)

  • Mike (unregistered) in reply to Satanicpuppy

    That story was about 100x better than TFA.

  • Meep (unregistered) in reply to DaveyDaveDave
    DaveyDaveDave:
    socknet:
    Absolutely no idea where you are quoting from..

    Have a look at the source - it's where the best bits of Remy's articles are to be found :)

    By best you mean "least worst?"

  • (cs)

    At least they were smart enough to not plug UPS #2 into itself.

    "Duh. It's an uninteruptable power source. Why can't it run itself?"

    The fact that no UPS can be plugged into itself is a power company conspiracy.
  • (cs)
    UPS #2:
    I can't handle that kind of load.
    That's what she said.
  • (cs)

    Alex should write a book. Or three. Look at the first sentence, and be awed. This article comes down to "We have 2 ups's; they were wired incorrectly; it's a miracle that even in good times the servers stayed up. Now they're really old, I asked for new ones, and didn't get a response". But like an artistically drawn flower, the story unfolds into a glory of dusty old cables, flashlights without batteries, and flickering lights.

    Good job Alex, keep up the good work!

  • Robb (unregistered)

    Wait, if I plug my UPS into another UPS that is like double backup!

    This can't fail!

  • Mojomonkeyfish (unregistered)

    Ugh, I wish this didn't hit so close to home. I inherited a server room with exactly the same setup, and I was left scratching my head as to why my old "boss" (I am a developer, he was the sysadmin... now I'm both) thought this was a good idea.

  • (cs) in reply to Someone
    Someone:
    Kind of silly if you think about it. A UPS powering a whole farm of servers (which are designed for two) and another UPS? Seems a bit of a ridiculous mistake to make.

    Is very easy to make such mistake! Sometime electrican person doesn't know what he is doing.

  • Rich the Engineer (unregistered)

    Any company with "management" stupid enough to ignore basic need for reliable power deserves to have all their servers crash and burn, unrepairably.

  • J Cooper (unregistered)

    Ok, so, isn't TRWTF this:

    "Hubert stumbled to the server room in the dark, guided only by the blinking lights of WiFi routers scattered throughout the building"......

    Wait, so the power is out, but the wifi routers are on? HTF? Were they powered by 9V batteries wired in series or something?

  • (cs) in reply to dogbrags
    dogbrags:
    Alex should write a book.
    Sorry Remy, YOU get full credit. And YOU should write a book or three.
  • Rich the Engineer (unregistered) in reply to Nagesh

    No electrical person would do something that stupid. There are electrical codes, like the NEC (National Electrical Code) for a reason! It's only when you get ignorant computer techies who think they know electricity because they can match a plug with a wall outlet, is when you get complete clusterf**ks like that.

  • Doozerboy (unregistered) in reply to socknet
    socknet:
    DaveyDaveDave:
    Since an inning in baseball is roughly equivalent to an over in cricket

    I know nothing about baseball, but I'm pretty sure you're wrong there... (http://en.wikipedia.org/wiki/Innings)

    Absolutely no idea where you are quoting from.. but it is much more interesting that UPS, so I'll go with it..

    an Innings in baseball is roughly equivalent to... shock.. an innings in cricket. I realise this may suprise a lot of people, so I'll give you some time to recover before making any further major revelations.

    I may have completely missed the point or joke in this post , but an over in cricket is not roughly anything like an innings in baseball.

  • source reader (unregistered) in reply to Doozerboy
    Doozerboy:
    socknet:
    DaveyDaveDave:
    Since an inning in baseball is roughly equivalent to an over in cricket

    I know nothing about baseball, but I'm pretty sure you're wrong there... (http://en.wikipedia.org/wiki/Innings)

    Absolutely no idea where you are quoting from.. but it is much more interesting that UPS, so I'll go with it..

    an Innings in baseball is roughly equivalent to... shock.. an innings in cricket. I realise this may suprise a lot of people, so I'll give you some time to recover before making any further major revelations.

    I may have completely missed the point or joke in this post , but an over in cricket is not roughly anything like an innings in baseball.

    The British are always wrong.

  • (cs) in reply to SCSimmons
    SCSimmons:
    Satanicpuppy:
    I got to the building, jumped out of the car, sprinted for the door. Took the stairs a landing at a time, and kicked into the server room to find...That a moron coworker had allowed some customer service reps to set up shop in the server room.

    (Pause for people to say...WHAT?)

    I call shenanigans on this story. I've worked in many different call centers, and I'm aware of the consistent level of professionalism shown by management at every company.

    There is no way you wouldn't have gotten fired for kicking the CSRs and their space heaters out of the server room. :)

    Fortunately customer service isn't our forte ;)

  • frist (unregistered) in reply to J Cooper
    J Cooper:
    Ok, so, isn't TRWTF this:

    "Hubert stumbled to the server room in the dark, guided only by the blinking lights of WiFi routers scattered throughout the building"......

    Wait, so the power is out, but the wifi routers are on? HTF? Were they powered by 9V batteries wired in series or something?

    power over Ethernet? (presumably from a switch inside the server room)

Leave a comment on “Reduhdant Power”

Log In or post as a guest

Replying to comment #:

« Return to Article