- Feature Articles
- CodeSOD
- Error'd
-
Forums
-
Other Articles
- Random Article
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
All bets that the person fiddling and "accidentally" pressing the emergency stop button was the same middle manger who was realizing he was going to have to pay bonuses. And he was pissed because the PBX stayed up and foiled his plan.
Admin
Were these buttons labeled? That idiot, was he blind, too?
Admin
Why would Burt have a need to rectify those marine batteries? Did he connect them in reverse to get an alternating current? <gd&r>
Admin
I suspect that Burt was looking for the word "inverter"...
Admin
I remember that if a PBX was sold or leased by The Phone Company, a backup battery was present, to make possible for people to use the internal phones in case of emergency to call emergency services. An if the battery was depleted the PBX was set to go in dumb mode and connect directly designated phones to external lines.
Modern PBX systems still have the connector for the 48V backup battery system, so it's possible that the sealed lead acid batteries were present as designed.
Admin
I'm old enough, and I've seen and experienced more than enough to feel sympathetic to Burt.
Also, if you give bonusses purely on some things which are easy to measure, you get employees who just focus on the measurable things, instead of doing what's best for the company.
Admin
If there was any justice in the world, that idiot didn't get a bonus that year. Want to bet that the idiot was a C-suite exec?
Admin
How timely. What if you had four backup banks, none of which were connected to an emergency shutoff?
https://www.datacenterdynamics.com/en/news/ovhcloud-fire-report-sbg2-data-center-had-wooden-ceilings-no-extinguisher-and-no-power-cut-out/
Highlight: 'According to the report firefighters on the scene found electrical arcs more than one meter long flashing around the door to the power room, and it took three hours to cut off the power supply because there was no universal cut-off."
Admin
Goodhart's law: "When a measure becomes a target, it ceases to be a good measure"
https://en.wikipedia.org/wiki/Goodhart%27s_law
Admin
Nothing in the writeup suggests there were any labels on or near the buttons. Never go with a simple solution when a daft one is available!
Admin
For me the red flags started with the statement of 100% uptime.
Are you seriously not going to apply security updates to a server that is in constant communication with the outside internet?
Admin
not sure if "rats nets of Ethernet cables" is accidental or not. Pretty good phrase though.
Admin
It's system uptime, not device uptime. You can update redundant servers in batches so that the system will never be down, just running with reduced capacity sometimes
Admin
Rolling updates. <showmyage>VAXClusters for the win!</showmyage>
Admin
A traditional PBX does not connect to the Internet. It is a self-contained computer that only switches copper wire connections on and off. If there were software updates, they'd be installed by floppy disk.
Admin
I understand the whole "break glass to pull lever" contraption concept, but these are always alongside ample labeling on what to do, how, when, etc
Admin
A rectifier would be required to convert mains power to DC for charging the batteries. If the PBX also ran off of DC (-48VDC is the most common) then no inverter would be required.
Admin
It still is the case, in various forms, today. For example, if you subscribe to Verizon FiOS, you can install a battery in the ONT's power supply. If mains power goes out, the battery will keep your phone up and running. When the battery capacity drops to about one hour of service, it will shut off the phones, but there's a button you can press to re-enable it until the battery completely runs out. This way you can always make critical calls (e.g. to the power company to report the outage).
Unfortunately, once the FCC stopped mandating these batteries, Verizon no longer provides it free of charge. The ONT's have the connection for the battery (as far as I know), but you need to pay extra to have one actually installed.
Admin
The modern way is not to get a bonus based on the actual performance, it's based on how close it is to some estimate given at the start of the year. And if you don't active what the expensive consultants predicted then it's your failure, rather than theirs.
Admin
Seems to me that (1) the PBX should have been located in its own room/closet apart from anything else, so an emergency server room "kill" doesn't take out the phone system, and (2) as already pointed out, PBXs have their own internal batteries and dead battery fail-over measures.
Admin
This way, we make sure that, next time someone presses the wrong button, the IT guys cannot be reached.
Admin
The company I work for (another couple of months) has a thing for 100% uptime. Nothing less is acceptable. People claiming just 9.9999999% are amateurs! Such as... IDK, AWS?
So last time a server went down the operator - who conveniently avoided training thanks to its company links - had a marvelous idea. Instead of reading the logs and troubleshoot the issue HE DELETED THE DATABASE.
He deleted the database and replaced it with any empty one. Because you know, databases are databases. They solve all the things. The system was down for more than a day, until I came back to office the next monday and restored it.
While I expressed disappointment about the handling, I noted our 99.6% uptime with no management, no dedicated infrastructure nor personel was "almost decent considered the situation". I then iterated the fact our technicians might have to be trained to properly use and maintain the new products.
Guess how were the yearly bonuses distributed.
Admin
It's a PBX. It can do updates without stopping.