• Brian Boorman (google)

    All bets that the person fiddling and "accidentally" pressing the emergency stop button was the same middle manger who was realizing he was going to have to pay bonuses. And he was pissed because the PBX stayed up and foiled his plan.

  • (nodebb)

    Were these buttons labeled? That idiot, was he blind, too?

  • Industrial Automation Engineer (unregistered)

    Why would Burt have a need to rectify those marine batteries? Did he connect them in reverse to get an alternating current? <gd&r>

  • (nodebb) in reply to Industrial Automation Engineer

    I suspect that Burt was looking for the word "inverter"...

  • MIKE (unregistered) in reply to Industrial Automation Engineer

    I remember that if a PBX was sold or leased by The Phone Company, a backup battery was present, to make possible for people to use the internal phones in case of emergency to call emergency services. An if the battery was depleted the PBX was set to go in dumb mode and connect directly designated phones to external lines.

    Modern PBX systems still have the connector for the 48V backup battery system, so it's possible that the sealed lead acid batteries were present as designed.

  • Abigail (unregistered)

    I'm old enough, and I've seen and experienced more than enough to feel sympathetic to Burt.

    Also, if you give bonusses purely on some things which are easy to measure, you get employees who just focus on the measurable things, instead of doing what's best for the company.

  • Andrew (unregistered) in reply to Mr. TA

    If there was any justice in the world, that idiot didn't get a bonus that year. Want to bet that the idiot was a C-suite exec?

  • WilliamF (unregistered)

    How timely. What if you had four backup banks, none of which were connected to an emergency shutoff?

    https://www.datacenterdynamics.com/en/news/ovhcloud-fire-report-sbg2-data-center-had-wooden-ceilings-no-extinguisher-and-no-power-cut-out/

    Highlight: 'According to the report firefighters on the scene found electrical arcs more than one meter long flashing around the door to the power room, and it took three hours to cut off the power supply because there was no universal cut-off."

  • trainbrain27 (unregistered) in reply to Abigail

    Goodhart's law: "When a measure becomes a target, it ceases to be a good measure"

    https://en.wikipedia.org/wiki/Goodhart%27s_law

  • Robin (unregistered) in reply to Mr. TA

    Nothing in the writeup suggests there were any labels on or near the buttons. Never go with a simple solution when a daft one is available!

  • ooOOooGa (unregistered)

    For me the red flags started with the statement of 100% uptime.

    Are you seriously not going to apply security updates to a server that is in constant communication with the outside internet?

  • (nodebb)

    not sure if "rats nets of Ethernet cables" is accidental or not. Pretty good phrase though.

  • Heinebold (unregistered) in reply to ooOOooGa

    It's system uptime, not device uptime. You can update redundant servers in batches so that the system will never be down, just running with reduced capacity sometimes

  • dpm (unregistered) in reply to ooOOooGa

    Rolling updates. <showmyage>VAXClusters for the win!</showmyage>

  • Ross Presser (unregistered) in reply to ooOOooGa

    A traditional PBX does not connect to the Internet. It is a self-contained computer that only switches copper wire connections on and off. If there were software updates, they'd be installed by floppy disk.

  • (nodebb) in reply to Robin

    I understand the whole "break glass to pull lever" contraption concept, but these are always alongside ample labeling on what to do, how, when, etc

  • (nodebb) in reply to Steve_The_Cynic

    A rectifier would be required to convert mains power to DC for charging the batteries. If the PBX also ran off of DC (-48VDC is the most common) then no inverter would be required.

  • David C. (unregistered) in reply to MIKE

    It still is the case, in various forms, today. For example, if you subscribe to Verizon FiOS, you can install a battery in the ONT's power supply. If mains power goes out, the battery will keep your phone up and running. When the battery capacity drops to about one hour of service, it will shut off the phones, but there's a button you can press to re-enable it until the battery completely runs out. This way you can always make critical calls (e.g. to the power company to report the outage).

    Unfortunately, once the FCC stopped mandating these batteries, Verizon no longer provides it free of charge. The ONT's have the connection for the battery (as far as I know), but you need to pay extra to have one actually installed.

  • The Shadow Knows (unregistered)

    The modern way is not to get a bonus based on the actual performance, it's based on how close it is to some estimate given at the start of the year. And if you don't active what the expensive consultants predicted then it's your failure, rather than theirs.

  • see sharp (unregistered)

    Seems to me that (1) the PBX should have been located in its own room/closet apart from anything else, so an emergency server room "kill" doesn't take out the phone system, and (2) as already pointed out, PBXs have their own internal batteries and dead battery fail-over measures.

  • Giulio (unregistered)

    This way, we make sure that, next time someone presses the wrong button, the IT guys cannot be reached.

  • UserK (unregistered)

    The company I work for (another couple of months) has a thing for 100% uptime. Nothing less is acceptable. People claiming just 9.9999999% are amateurs! Such as... IDK, AWS?

    So last time a server went down the operator - who conveniently avoided training thanks to its company links - had a marvelous idea. Instead of reading the logs and troubleshoot the issue HE DELETED THE DATABASE.

    He deleted the database and replaced it with any empty one. Because you know, databases are databases. They solve all the things. The system was down for more than a day, until I came back to office the next monday and restored it.

    While I expressed disappointment about the handling, I noted our 99.6% uptime with no management, no dedicated infrastructure nor personel was "almost decent considered the situation". I then iterated the fact our technicians might have to be trained to properly use and maintain the new products.

    Guess how were the yearly bonuses distributed.

  • (nodebb) in reply to ooOOooGa

    It's a PBX. It can do updates without stopping.

  • David C. (unregistered) in reply to Planar
    Comment held for moderation.
  • jay (unregistered)
    Comment held for moderation.

Leave a comment on “Perfect Uptime”

Log In or post as a guest

Replying to comment #:

« Return to Article