• jimicus (unregistered) in reply to s
    10 minutes for decision from the boss, 5 minutes to find an employee with a pickup truck, half a hour drive to the store, 5 minutes to find it and get it to the car, 10 minutes of getting the invoice, another half a hour to get it back to the site, with 5-minute stop to buy fuel. Setting up a portable generator (gasoline, not diesel) takes maybe 15 minutes. Installing it is as easy as plugging the UPS into it.

    You are assuming that the server room in question was small enough that it could be powered off the kind of generator one can easily purchase at retail and stick in the back of a car. I've seen plenty where that would not be an option.

  • (cs)

    We moved our entire datacenter to a new building. We weren't a huge shop, but I would estimate about 100 racks and 800 servers. We only had a weekend window, so we planned for months. Mapping out where each rack will go, making sure there is power (the CORRECT outlet and voltage), pulling a fiber for each switch, making sure a tile is cut so you can access the power, etc.

    Moving day went surprisingly well. The one server that was "my baby" was our webserver (as I was the only developer and the only one who was certified in Linux). I mentioned several times in these planning meetings that the internal webserver would obviously be turned off for the move. "Yes, we know, everything is going to be turned off so it doesn't matter." I replied that our entire customer base and the other half of our team (located in another state) rely on the website for day to day activities. "Bah! It's just development, they can deal for a few hours."

    The webserver came back up just fine, but it was interfacing with a shared Oracle server which took several days to get back up. Someone shut it down by pulling the breaker on the rack (not intentionally). So they spent a while indexing and repairing Solaris and Oracle.

    My manager got reamed by HIS manager, which trickled down to me. For something I had no control over and which I tried averting.

  • poindexter (unregistered) in reply to FredSaw
    FredSaw:
    Rugger fan:
    Wow!!! Stop the presses!!! Call the media!!!
    Um... if you can stop the presses, you are the media.
    Or a evil .gov?
  • NeoMojo (unregistered) in reply to Sgt. Preston

    complex mutable banana

  • sewiv (unregistered) in reply to Notta Noob
    Notta Noob:
    "Your faith in your [generator] will be your[ downfall]." (loosely quoting RotJ) It's just super that you have an ATS that allows for a 10-second cutover, quite fantastic, really. Now what happens when you have a generator failure due to, well, pretty much pick any mechanical reason a generator can fail? Do you think an hour will give you enough time to: a) find a replacement generator; b) hire an electrician to re-wire it into your building EP grid; c) fire-up & test the replacement before cutting it over?

    Since the "replacement" is also called our secondary generator, is already on site and wired in to the secondary ATS, and will cut in if the primary fails (again, 10 seconds), both generators get exercised weekly and get quarterly maintenance and semi-annual load tests, AND we have 36 hours of fuel on site for each generator, with priority contracts for fuel delivery, yeah, I have pretty good faith in my systems.

    There comes a point when you can be too paranoid about power, but I haven't gotten there yet. I trust generators more than UPSes, though, so I'd rather get my runtime from them. (All my UPSes are redundant as well.)

    I tried to convince the PTB that the generators should have multiple fuel sources (one diesel, one natural gas), but they didn't go for that, since we don't have gas on site.

    They also weren't cool with getting a second feed from a different power company, because there isn't one around here.

    All in all, given the budget limitations, we're doing okay.

  • sewiv (unregistered) in reply to operagost
    operagost:
    You can get a diesel generator installed in eight hours? Quite fantastic, really.

    Actually, you can. Portable generators on trailers are available, depending on your size requirements. It helps to pre-negotiate this with your generator maintenance company.

  • (cs) in reply to sewiv

    Portable generators on trailers are available, depending on your size requirements.

    They exist, yes. But unless you already have one parked on-site when the power goes out, good luck getting one.

  • (cs) in reply to Rich
    Rich:
    FredSaw:
    Rugger fan:
    Wow!!! Stop the presses!!! Call the media!!!
    Um... if you can stop the presses, you are the media.

    A medium, surely?

    Ah, the old singular/plural discussion is still alive. I was using the collective "you"--as in, you the company, consistiing of the guy giving the "Stop the presses!" order, the pressmen who follow that order, the reporters who write up the breaking news that is more important than what was on the presses, and so forth.

    Besides, spirits from beyond told me it's "media".

  • snoofle (unregistered) in reply to Notta Noob
    Notta Noob:
    sewiv:
    Ridiculous. If you have a generator and a UPS (with 8 HOURS of runtime, which is also ridiculous). Our UPSes have about an hour of runtime, and seldom use more than 10 seconds of it, since that's how long it takes for the ATS to start the generator, test that it's receiving good power from the generator, and switch over to the generator. When the power comes back, the ATS waits 15 minutes to make sure the power is stable, then switches back and shuts down the generator (after a cooling-off period).
    "Your faith in your [generator] will be your[ downfall]." (loosely quoting RotJ) It's just super that you have an ATS that allows for a 10-second cutover, quite fantastic, really. Now what happens when you have a generator failure due to, well, pretty much pick any mechanical reason a generator can fail? Do you think an hour will give you enough time to: a) find a replacement generator; b) hire an electrician to re-wire it into your building EP grid; c) fire-up & test the replacement before cutting it over?
    sewiv:
    That's just how it's done. Anyone who installs a system that does otherwise should be fired/sued into non-existence. The ATS is the cheapest part of our setup.
    About the only part of this statement I agree with is the ATS part and related liability. I can tell you that "how it's done" is entirely dependant on your company's continuity objectives.

    As the IT Manager for a scheduled airline, my objective was keep things running as long as possible. So yes, we had our key systems connected to the building EP Grid, with a 5 second transfer to gen-power at loss of building mains, on top of that we had a UPS monstrosity that would run all of our servers, network gear, phones, and key workstations for 8 hours. At about the 4 hour mark on the UPS, our managed power bars would start shutting off power to all non-essential peripheral equipment. At the 2 hour mark, non-essential servers would begin the shutdown process. At the 1 hour mark, only two servers, two switches, two routers, one workstation and the phone system would remain powered. At the half hour mark, the phone system would have power removed and work off of its internal batteries (good for 2 hours) and the UPS would run down to zero.

    Overkill? Well, depends on whether you'd like to see the company monitoring your aircraft have some idea of where those aircraft are and be able to communicate with them. The process outlined above could theoretically extend the time of power delivery from 8 hours to just under 24. As for us, we never experienced an outage beyond 6 hours, but the 6 hour outage we did experience was due to, of all things, a generator failure while the building was blacked out :)

    Is it ironic or just entertaining that the captcha for this post is 'alarm'?

    Generators (or any mechanical device) rarely fail while sitting around, waiting to be powered up; they only fail when they are being used, which in this case, is when you can least afford it.

    I once had the privilege of working for a large place where management actually got the concept of redundancy. They had multiple parallel UPS's good for a couple of hours to run the entire building (including data center). Backing them up were 3 huge kerosene fueled generators that could run as long as there was fuel in the tanks. There were monthly cutover tests (while folks were working), with the idea being that if something didn't work seamlessly, we wanted lots of folks to scream so we'd know about it and have the chance to fix it.

    Good times.

  • JEH (unregistered) in reply to s
    s:
    10 minutes for decision from the boss, 5 minutes to find an employee with a pickup truck, half a hour drive to the store, 5 minutes to find it and get it to the car, 10 minutes of getting the invoice, another half a hour to get it back to the site, with 5-minute stop to buy fuel. Setting up a portable generator (gasoline, not diesel) takes maybe 15 minutes. Installing it is as easy as plugging the UPS into it.

    This is all providing you get a blank cheque from the boss. Otherwise, purchase approval can take up to a month.

    It's not a hi-tech solution, but if you want it fast and improvise, you don't have time for hi-tech.

    You don't get at 400 kVA gasoline powered generator at a nearby store - and 400 kVA is not that much...

    The generator setup at a facility should also be N+1 - as far as I know we have 3 generators (in the 600-800 kVA class) where I work (telco) and can be run on 2 - and the generators are tested every 2 months or so (and for the record, testing also uses diesel...).

  • AscendingDragon (unregistered) in reply to Notta Noob

    Notta:

    It's called operational control authority, and your flight superintendents are required by FAR 121 regulations to maintain it. If you lose your systems, and aircraft are operating, you will see a scramble in the operations center that will make Apollo 13 look like a kindergarten play.

    I've been in a regional airline's ops center when we lost all of our computers. We were back to a fax machine powered by an extention cord routed to another office. Talk about batshit crazy, we were using our personal cell phones to file flight plans with the FAA, and faxing flight releases to airside ops so that we could still operate. We had to cancel our operations out of two of the most popular hubs in the world, and cost the company millions because of those cancelations.

    That Monday morning operations meeting was not a good one.

  • Zygo (unregistered) in reply to KG2V
    KG2V:
    Yep - I was going to say - that's what 1 hard wired POTS line is for. Standard disaster recovery stuff

    The best thing about those is you can draw almost 3 watts out of them. Even more power is available if you can arrange for the lines to ring. ;-)

  • Zygo (unregistered) in reply to s
    s:
    10 minutes for decision from the boss, 5 minutes to find an employee with a pickup truck, half a hour drive to the store, 5 minutes to find it and get it to the car, 10 minutes of getting the invoice, another half a hour to get it back to the site, with 5-minute stop to buy fuel. Setting up a portable generator (gasoline, not diesel) takes maybe 15 minutes. Installing it is as easy as plugging the UPS into it.

    This is all providing you get a blank cheque from the boss. Otherwise, purchase approval can take up to a month.

    It's not a hi-tech solution, but if you want it fast and improvise, you don't have time for hi-tech.

    That would be enough power for the alerts server and the PBX. After you get those up and running, they can send the right pages to the right IT people so the big diesel generator can be started and the other 600 systems can be brought online. ;-)

    Also, have you ever actually been shopping for a portable gasoline generator during a non-trivial power outage? Just look for the rows of dusty, empty shelves with generator-shaped dust-free outlines in the dust, then travel back in time to 5 minutes before the lights went out and be one of the people who bought a generator before you knew you needed one.

  • James (unregistered) in reply to KG2V
    KG2V:
    Yep - I was going to say - that's what 1 hard wired POTS line is for. Standard disaster recovery stuff

    Of course, had they done that, the punchline of the WTF would instead have been something like:

    "Finally, we checked our disaster recovery plan, and noticed that while the plan specified that the emergency paging system should be hooked to a POTS line, it was actually connected through our PBX. When we finally traced the wire, we found a sticky note on which was scrawled 'This was in an old wall jack for some reason. Somebody must have missed it during the last migration. I plugged it in with the rest of the PBX lines -- people should pay more attention.' Nobody would admit that it was their handwriting."

  • AdT (unregistered)

    Did Phil turn the power off while the machine was running fsck? Now that's fscking brillant! Although a good journaling FS would probably get over it.

  • Notta Noob (unregistered) in reply to operagost
    operagost:
    You can get a diesel generator installed in eight hours? Quite fantastic, really.
    You can when you work for an airline and have two trailer-mounted GPU's that output a voltage/amperage that your UPS is happy to accept available not even 4Km's away :)

    That and in the major center where we operated there are at least two 24/7 industrial equipment rental places that can have a generator of considerable size on site in <4 hours (never had to test that service so I can't tell you what reality looked like). Was ridiculously expensive, but cheaper than the alternative.

    Believe me, when grounded aircraft can start costing you $$$$ after just a few short hours (duty-days, airport TOD noise-restrictions, etc. resulting in pax accommodation requirements, etc.), you work out your contingencies WELL ahead of time.

  • Notta Noob (unregistered) in reply to AscendingDragon
    AscendingDragon:
    Notta:

    It's called operational control authority, and your flight superintendents are required by FAR 121 regulations to maintain it. If you lose your systems, and aircraft are operating, you will see a scramble in the operations center that will make Apollo 13 look like a kindergarten play.

    I've been in a regional airline's ops center when we lost all of our computers. We were back to a fax machine powered by an extention cord routed to another office. Talk about batshit crazy, we were using our personal cell phones to file flight plans with the FAA, and faxing flight releases to airside ops so that we could still operate. We had to cancel our operations out of two of the most popular hubs in the world, and cost the company millions because of those cancelations.

    That Monday morning operations meeting was not a good one.

    Yep. Seen days like that. Like the day a third-party company went through every single inbound comms cable to the building with a portable cut-off saw (how do you realistically plan for that?!?) while they were 'cleaning up' old cabling in the main comms room (which they were under the impression was an un-used wiring center). Our Dispatchers had to relocate to our Emergency Ops Center, switch to fax/cell phone and use SkyPlan to file the remainder of flights for the day. There were some very unhappy people in our office for quite some time... only delayed a few flights briefly though so the cost was relatively low.

  • (cs)

    Is it true that Solaris is unhappy about having the plug pulled? Old mainframes could tolerate that without any problems. I thought that journaling filesystems would be unaffected by sudden loss of power. Databases are supposed to have "ACID": atomicity, consistency, isolation and durability. ACID doesn't let them survive disk corruption, but a journaling filesystem is supposed to prevent that. I know that with a Linux box and ReiserFS it hardly matters whether you shut it down or push the button on it.

    I have switched to Ubuntu, which normally installs on Ext3, which is frustrating because it wants to fsck every 23rd boot. Seems like the days of fsck should have been over by now. Especially frustrating on a laptop.

    Shame on Mailboxes Etc in Beverly Hills

  • (cs) in reply to probabilities
    probabilities:
    operagost:
    Notta Noob:
    "Your faith in your [generator] will be your[ downfall]." (loosely quoting RotJ) It's just super that you have an ATS that allows for a 10-second cutover, quite fantastic, really. Now what happens when you have a generator failure due to, well, pretty much pick any mechanical reason a generator can fail? Do you think an hour will give you enough time to: a) find a replacement generator; b) hire an electrician to re-wire it into your building EP grid; c) fire-up & test the replacement before cutting it over?

    You can get a diesel generator installed in eight hours? Quite fantastic, really.

    At least you have 800% more opportunities to get it.

    If you have a small setup (a couple of racks on standard power), gas generators can be purchased at a local hardware store and are usually enough to keep things running for a while

    Although....if you were really keen (and had a larger setup), you'd have DRP and/or BC (Disaster Recovery process and Business Continuity) plan implemented so that your main datacenter could literally disappear and within the hour your business would be back online from another location.

  • (cs) in reply to Pingmaster
    probabilities:
    operagost:
    You can get a diesel generator installed in eight hours? Quite fantastic, really.
    At least you have 800% more opportunities to get it.

    You can get just about anything quickly if you're willing to pay enough for it.

    Pay triple for the generator and promise some electrician triple overtime and you'll have power.

    We only have one generator, but we have a slot on the outside of the generator building that you can plug another generator into if the main generator fails or we need more power.

  • Daniel Smedegaard Buus (unregistered)
    Solaris does not take kindly to having the lights go out unexpectedly
    A perfect pun if there ever was one :)

Leave a comment on “Paging Dr. UPS”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article