• Steve (unregistered)

    At one very well-known manufacturer of high-priced printers and computers, the big red button in the server room was directly behind the door (which opened inwards). It lived there quite peacefully until one day, when somebody entered the server room in a rush, and suddenly everything went quiet. The button was subsequently moved...

    -- Steve

  • Stan (unregistered) in reply to Steve

    In the early days of PC ATs one group filled a cubicle with servers and printers. Cubes were only wired with two outlets for 3270 terminals, so they had the electrician put a line around the cube with outlets every foot or so. And ran that line through an ordinary household lightswitch that made it convenient to shut the whole mess down when you went home and fire it back up the next morning. (!) I guess the spike from that switch was not as clean as the built-in power supplies might have been because they had many mysterious failures ... mother boards, graphics boards, Six Packs (anybody else buy one of those?).

    Also in those days our building manager was a mean spirited Grinch lookalike, a real hardass on anybody who slipped up. Or didn't. He escorted an electrician back into the computer bays (roughly football field sized) to personally turn off the power on the empty bay so they could install equipment. He grabbed the wrong giant lever and turned off the active bay. Ooops. Some of the mainframes were so old and powered down so rarely that it was a real crap shoot whether or not they'd ever come back up ... if anybody remembered the cold start sequence. He got lucky, we only lost a day.

  • jweller (unregistered) in reply to Jamie

    I work in a place where there is a BRB, right next to the door, that opens the door. So I guess I can understand why people might think the BRB opens the door. After years of "don't touch the BRB" it's sort of a WTF to actually use it.

    At home, I became completely convinced that my PC had a bad power supply. That is until I correlated spontaneous resets and exuberant tail-wagging around the reset button. I have since constructed a "molly guard".

  • sam (unregistered) in reply to Shareware
    Comment held for moderation.
  • jayh (unregistered)

    When we did an install of new UPS, after all the equipment was shut down, we drew lots to see who had the honor of finally hitting the BRB. Alas, it wasn't me.

    BTW--- the hell with bring your kid to work day, I am lobbying for bring your DOG to work day.

  • bofh69 (unregistered) in reply to Nick Holland
    I managed to convince our people that the likelihood of something WRONG happening with the BRB was much greater than the likelihood of it being needed and actually HELPING.

    I used to work at the same facility that aquatone282 mentioned above, and once a year we had to take the BRB apart, disconnect the Halon tanks from the system, and press the BRB to make sure that it really worked. I was so worried that I was going to miss a power cutoff or Halon connection that I triple checked everything before pushing the button. I'm glad to say that everything worked and there were not accidental power outages or Halon discharges. Then we put it back together and waited another year.

  • Barton (unregistered)

    Look up "Molly Frob" in the jargon file.

  • William (unregistered)

    I had a moment similar to this a few years back when I was at a web hosting company who's name I want say. We had about 20 servers, near Lucas Arts ILM render farm which at that time was in Exodus in downtown Los Angeles.

    I had been planning to move the 20 servers and after I moved the last customer I called the datacenter and asked the person on duty to kill the power to our cage, and gave him our name. He said he would "call me back on his cell at the cage" for "security purposes"

    When he called back he was yelling into the phone (noise datacenter) asking which cage was ours, I kept telling him.. Then I listened as it got really, really quiet. He was like. All done now. Then he asked me why Lucas would be shutting down all those servers that were just installed, and if I wanted them powered back up.

    Needless to say, I almost DIED laughing and crying, I told him to turn the power back on, and made every attempt to call ILM, he said he would call.

    I still tear up to this day thinking about the security guard calling ILM to let them know he accidentally turned off there 200+ server farm.

    In case you are wondering, (I shouldn't be saying this) but if you know what datacenter a company is in, and the datacenter number, there probably a 95% chance you can call it WITH NO PASSWORD and get them to turn off or cycle the power.. VERY SCARY.

  • William (unregistered)

    I do have to add one more comment. Back when "MCI Worldcom" was around, I toured their building in Columbus, Ohio. There main OC-192? Whatever connection to the west coast went through this small room. Normally you aren't even allowed to go back this far, but I was with my friend and he insisted on showing me the room. In this room, there were two guys sitting in Lab Coats, browsing on their laptops, just goofing off. The room was about the size of a large closet and had a small bulletproof glass window to see them through, it had double doors on either side. But inside that room, were two red phones, and a small center island table with two large switches. My friend told me that turning those switches off would complete turn off the internet and about 90% of all telephone service.

    Impressive site to see. Apparently one of the phones went straight to a government call center so they could call in if they had reason to turn off the "net" or communications.

    very strange.

  • Tony Taylor (unregistered)

    WTF, no kids allowed. never mind how much the 3 milliseconds cost the company

  • John (unregistered) in reply to William

    I too have a MCI/Worldcom story. I was just hired as a contractor to work on some software and was given a cubicle to work in. I go to the cubicle and there is a desktop running with the default redhat screensaver.

    "Cool" I thought, I was afraid I would have to develop on Windows NT 4.0, the thought of doing linux work on a linux desktop made me more comfortable with the whole project, which I already had lost confidence that it was a professional project and began to realize it was going to be a massive hack job.

    So I move the mouse, username/password was the prompt, I look under the keyboard/mousepad/mouse, no username or password was given to me. "Weird", I thought, I'll just wait for the IT guy, no problem.

    The IT guy shows up, with a new (as in new to me, it was actually 3 years old at the time) desktop, saying "Here is your desktop, oh hey, what's that computer for?". I say, "oh, I don't know, should I move it?". "Yup", was the answer.

    So I can't remember how I turned it off, I might have just pulled the plug. I set up the new computer, boot into windows, get email set up, start downloading Vim etc.

    Around an hour later, some guy runs up to me, says, "what happened to that computer?!?!?!?".

    Me, "oh, its under the desk over there."

    Him, "That was the production machine for the Texas Project!!!"

  • Jorge (unregistered)

    yeeeears ago, I worked with a development team, using NCR Towers (Unix) multiusers systems, the good old days of C, curses, text terminals, Unify databases. One day, a programming tech was hired, fresh from school. He had worked only with PC's (MS-DOS back then, Windows 1.1 was not introduced until one year later). So, he was going at best he could with Unix/C/Curses. One day, a program he was testing/debugging, crashed. The elder of us, will remember that if a Curses app. crashed, the terminal was usually left in "cooked" mode, so you were left with no echo, frozen cursor, etc. Being a DOS user, he knew that the solution for a frozen computer is Alt-Ctrl-Del right? ... but obviously it did not work. And if that did not work, what does a PC user does next? of course!! power off/power on, right? right! so, (all this without asking no help or advice from any of his mates sitting right next to him) he went to the computer room, and promptly pressed the power switch on a NCR Tower server, where 20 more users were connected. AND, the same elders will remember, the boot time for a NORMAL reboot on a Unix system back then was 15/20 minutes. After an abnormal shutdown, you have to add another 40 minutes for the fsck run. And the Unify database recovery. And the 20 irate coworkers whose work was lost (some were lucky with vi -r). The development servers were in a simple glass walled room, without all the fancy security controls used in the production datacenter. I think that since then, he was never again seen even in the same building with a power switch.

  • Scotty (unregistered) in reply to big zig
    big zig:
    Luckily it was only a data center. Imagine this being a nuclear reactor, so the power grid in California. People do some dumb things to impress kids during those visits.
    What, you think that never happened? I used to work at a nuclear power station, and on occasion would enter the Control Room, the inner sanctum where the reactor operators hang out with all those huge panels full of dials, buttons, and switches. On one panel there are TWO big red buttons, spaced about 8 inches apart so that it is not possible to hit both with one hand. Hitting both simultaneously will shut down the reactor by causing the control rods to be inserted, which contain a neutron absorber. This is a big deal, because in addition to the 750 megawatts or so of reactor output dropping off the electrical grid, the reactor can't be restarted for the better part of a day, once the steady-state of neutron creation/absorbtion has been disturbed. Neutron-absorbing by-products of the fission spike in level after the reaction stops and have to decay away first. Sadly for one operator who leaned back in his chair and tipped over, the buttons were not farther apart than his hard-hat was wide.
  • sim (unregistered)

    We once had a auditor (from our Blue Chip parent company's IT department - an organisation comprised almost entirely of managers) walk around our data centre, look closely at a one of our multi-CPU Sun boxen and poke the power button with a pen.

    Everyone standing in the room stared at him with a What the F...? expression, whilst it slowly dawned on him what he'd done.

    We never did tell him it was test server that wasn't being used...

  • Mike (unregistered)

    It only takes once 'incident' for those buttons to get the plexiglass treatment.

    In a facility I worked in, the incident was a network guy doing work above the ceiling panels, and backing into the button, while crawling out. The buttons were emergency power cut-offs for the entire server room.

    Within a week every button was covered.

  • My Name (unregistered) in reply to Fredric
    Fredric:
    I just hope the pentagon doesn't have bring-your-kids-to-work-day...

    perhaps 9/11 WAS a coverup?

  • darwin (unregistered) in reply to Fredric
    Fredric:
    I just hope the pentagon doesn't have bring-your-kids-to-work-day...

    Bring-your-kids-to-work day at the White House:

    Jenna: Daddy, what does this button do?

    [click]

  • darwin (unregistered) in reply to b
    b:
    David:
    Brian:
    At least it didn't launch a missle

    I think you mean missile.

    why don't you keep completely irrelevant and childish observations like that to your self?

    I think you mean yourself.

  • elKodos (unregistered)

    At a place I used to work many years ago, there was a BRS on the processor, and another on the wall beside the exit from the climate-controlled room. The one on the wall was for the air-conditioner. You know, in case of a runaway cooling event.

  • Jerry Steinberg, Founding Non-Father of NO KIDDING! (unregistered)
    Comment held for moderation.
  • TheQuux (unregistered) in reply to sim

    [Stabs power button on Ultra10 next to desk]

    Nope... power still on. Maybe I set up power management correctly...

    "STOP-A power-off" did the trick though... but that's not likely to be something you type accidentally.

    The power switch on the back would have been more likely to cause a problem, but that's a different story.

    [Tries the other Ultra10's power button] That one worked... but it went through standard shutdown (judging from hard disk activity)

  • Will (unregistered) in reply to fmtaylor
    fmtaylor:
    Guess they never heard of a molly-guard. Google it of you are unsure....
    I think every company has their own name we call ours a BOB Guard
  • Tom (unregistered)

    Oh the red button there... don't ever, EVER touch the red button.

    (from Men In Black)

  • 8bitwizard (unregistered)

    I have a "big button" incident to relate. It's not quite the same fuckup scale (it took less than a minute to reset, rather than 72 hours), but it still startled me. A few years ago I was visiting a major university here in Texas. I walked out of the parking garage through an entrance and noticed that there was some kind of sensor at the top of that metal arch over the entrance.

    You know, the arch that shows the max clearance and has a bar on hangers to bump into your vehicle so you can know you're about to shave off the top of your big truck? Well, there was also an obvious sensor beam thingy up there at the top.

    The temptation was too much, so I jumped up and waved my hand through the beam. Suddnely, a FSCKING LOUD alarm went off! See, it wasn't enough for them to have just a big metal bar to keep idiot truckers and RV'ers out of their parking garage, they wanted to be absolutely certain by making lots of noise and flashy lights.

    I sheepishly wandered off, and after about 30 seconds the alarm turned off automatically.

  • icelava (cs)

    How old is the CIO? 21?

  • Peter (unregistered) in reply to danixdefcon5
    Comment held for moderation.
  • Daniele Muscetta (unregistered)
    Comment held for moderation.
  • dean (unregistered)

    i have three children ii took them to work with me and they were jumping every where i did not know what to do i worked at a phone shop and when i went to sell a phone my kids had locked thm thelves in the phone closet and then the manager came in and unlock the stock room to find that they had broken all of the phones i got fired

  • qovsijxtb mgow (unregistered)

    azor fiqyeugs sbdf xceqadzn arwviu zqvphn ujnogmhti

  • Eh (unregistered)

    Not quite an IT Fuckup, but you'll get the idea...

    Background: in some industrial environments with flammable materials, it is neither safe nor cost-effective to use Halon as a gaseous fire supressant agent. Therefore, some use C02...

    I was working at just such a facility that happened to paint car bodies - the paint plants (there were 2) each had their own mix rooms and bulk storage areas (paint & solvent were stored in the bulk storage which was connect to the mix/pump rooms which then ran lines out to the actual paint booth areas).

    Each of these bulk storage/mix rooms was protected by a 20,000 (yes - 20 THOUSAND) pound tank of liquid C02 outside of the plants. The idea being, should a fire break out and get out of control, the sudden dumping of all the C02 would serve several purposes - namely entinguishing the fire and dramatically lowering the temperature to prevent a BLEVE event from occuring (Wiki BLEVE if you're not familiar with the term - those of us who've fought fires are). Now you're familiar with the setup.

    One day I'm making my rounds through the bulk storage/mix room areas and am watching this electrician do some maintenance work. Suddenly I see him reach for the BIG RED BUTTON...

    All I can say is thank GOD for the delay. Otherwise, I'd be dead and so would he.

  • Argent Stonecutter (unregistered)

    Back before they nerfed some of the calls in Second Life it was possible to launch people millions of kilometers in the air, so high that floating point rounding errors did the most AMAZING things to your avatar.

    I made an object called "don't touch me". It was a little red button, not a big one, with about 128 scripts all doing this "launch into orbit" trick if you touched it. As a safety feature, it asked if you really meant to touch it, nothing happened accidentally, you had to decide that you were really going to touch it.

    When you did that, you basically disappeared. Sometimes you'd see yourself bouncing around on the screen as the program tried to keep you in sight despite the rounding errors. You often had to log in again to get back to a sane state.

    It's amazing how many people still just had to do it.

  • Who Me? (unregistered) in reply to Jethris

    And if said electrician was ever heard from again, it was only after a VERRRRY long period of "questioning", I'm certain.

  • John (unregistered)

    Years ago I worked in a computer lab with big red buttons placed every thirty feet or so along the wall. It got to the point that none of us really thought much about them.

    Then came the day when the plumber turned off the four inch feeder pipe and proceeded to take a sledge hammer to the rusted section, without determining the direction of the waterflow... I opened the door on my way to a meeting to see a small wall of water rolling down the hall towards the lab, and the dozens of workstations we had sitting on the floor.

    I slammed the door shut and spiked that red button while ordering all of my co-workers to start putting workstations on boxes chairs and the tables as fast as we could. With water pouring under the door we managed to get all of them up out of the way before the room flooded. We lost a few O'Reilly X-Windows books, but no hardware or people thanks to the red button to turned off the power to all of the power strips we also had on the floor.

  • Larry Osterman (unregistered)
    Comment held for moderation.
  • f0urtyfive (unregistered)

    I worked a ski lift for the past 6 months, it had plenty of emergency stop (and normal stop) switches on it, if I can recall correctly, the one I normally worked had, 6 emergency stops and 5 normal stops. Needless to say, they tend to get in the way when their is more than 1 person around and they were bumped quite a few times accidentally. Starting it back up took about 60 seconds though.

  • far2bored (unregistered)

    We've got a button in our data center. We refer to it as the Change of Career Button. Of course I work for a porn company, so there really isn't much turnout for bring your kid to work day.

  • antigrammarnazi (unregistered) in reply to Robert
    Robert:
    That should be "years" not "ywars"

    yeah you think? Are you the only one stupid enough to have to have typos pointed out to him?

  • Not Going To Tell You (unregistered)
    Comment held for moderation.
  • t (unregistered)

    Our big red button fires CO2 into the data center. That would have taught the kid valuable lessons in both button-pressing and biology.

  • Years Ago... (unregistered) in reply to Scoldog
    Comment held for moderation.
  • Craig (unregistered)

    I worked in a large datacenter in the UK a few years ago. Next to the exits of the server room were the big red buttons of mass destruction.

    Our datacenter manager was moving some new kit into the server room on a trolley, and as he opened the door he reached around inside to steady himself and slammed the big red button.

    The whole building went very quiet very quick and emergency lights came on. Oh to have had a camera to capture his face when he did it!!!

  • Craig (unregistered)

    I worked in a large datacenter in the UK a few years ago. Next to the exits of the server room were the big red buttons of mass destruction.

    Our datacenter manager was moving some new kit into the server room on a trolley, and as he opened the door he reached around inside to steady himself and slammed the big red button.

    The whole building went very quiet very quick and emergency lights came on. Oh to have had a camera to capture his face when he did it!!!

  • Craig (unregistered)

    I worked in a large datacenter in the UK a few years ago. Next to the exits of the server room were the big red buttons of mass destruction.

    Our datacenter manager was moving some new kit into the server room on a trolley, and as he opened the door he reached around inside to steady himself and slammed the big red button.

    The whole building went very quiet very quick and emergency lights came on. Oh to have had a camera to capture his face when he did it!!!

  • FH (unregistered)

    Oh, for heavens sake, folks. You are not a real geek unless your kids have heard the mantra "No touching buttons, knobs or dials" from the moment they can first inch their way across the floor towards them.

  • Steve (unregistered)

    I was a shiftwork operator running the biggest ICL Mainframe installation in the southern hemisphere (a dual-node 3980 and a 3950). We had a Big Red Stop button mounted quite high up on the wall near the airlock entrance. We'd run out of space to stock lineflow (continuous paper) boxes in the printer room so decided to put some boxes in the main machine-room. My shiftworking partner was doing a conan impression by carrying in three boxes stacked on top of each other (they are heavy!) and misbalanced managing to smack the Big Red Button. Darkness and silence except for faint ticking sounds as the machines cooled down.

  • Hognoxious (unregistered) in reply to David
    David:
    Brian:
    At least it didn't launch a missle
    I think you mean missile.
    He's probably American and writing it how he pronounces it.
  • Anthony (unregistered) in reply to Bob

    I boought my son into work many years ago. I think he was about 3 or 4. I was in the PC room and turned my back for a minute. He had gone through the menus, just pressing enter and "Format the C drive" was the last one at the end of the chain!

    After the disk was restored, the menu got changed!

  • dick (unregistered)

    Yeah, they have big red buttons on ventilators, CT scanners, MRI machines and anesthesia machines. The one on the MRI is particularly impressive when it vents all the liquid helium to the atmosphere...

  • starjax (unregistered) in reply to Jno
    Jno:
    The genius who installed our secure computer room (Faraday cage, with air-lock-stylee double doors interlocked by pneumatics) could do no better than put the Emergency Power Off button on the wall right next to the Open The Damn Door button. We too got a perspex box after the fifth time the lights went out ...

    I know the feeling. Try having a big red button that not only powers down the data center, but the entire campus. This was a DC designed in case of tornado, nuclear attack, or whatever. I got stuck in a remote office for 2 hours till they were able to bring everything back up.

  • SoldierDude (unregistered)

    My Big Red Button Story has nothing to do with kids at work. It has to do with stupid people putting two Big Red Buttons next to each other. It was a data center at one of our top military installations. The Big Red Button that turns on the flashing red lights to announce "Uncleared Personnel" was placed next to the Big Red Emergency Power Off Button. Sure enough, one day some one powered off the entire data center when they wanted to turn on the flashing lights. It only took me a day to recover my UNIX servers. But it was three days before the IBM 390 was back to normal.

Leave a comment on “The Big Red Button”

Log In or post as a guest

Replying to comment #:

« Return to Article