- Feature Articles
- CodeSOD
- Error'd
-
Forums
-
Other Articles
- Random Article
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
At one very well-known manufacturer of high-priced printers and computers, the big red button in the server room was directly behind the door (which opened inwards). It lived there quite peacefully until one day, when somebody entered the server room in a rush, and suddenly everything went quiet. The button was subsequently moved...
-- Steve
Admin
In the early days of PC ATs one group filled a cubicle with servers and printers. Cubes were only wired with two outlets for 3270 terminals, so they had the electrician put a line around the cube with outlets every foot or so. And ran that line through an ordinary household lightswitch that made it convenient to shut the whole mess down when you went home and fire it back up the next morning. (!) I guess the spike from that switch was not as clean as the built-in power supplies might have been because they had many mysterious failures ... mother boards, graphics boards, Six Packs (anybody else buy one of those?).
Also in those days our building manager was a mean spirited Grinch lookalike, a real hardass on anybody who slipped up. Or didn't. He escorted an electrician back into the computer bays (roughly football field sized) to personally turn off the power on the empty bay so they could install equipment. He grabbed the wrong giant lever and turned off the active bay. Ooops. Some of the mainframes were so old and powered down so rarely that it was a real crap shoot whether or not they'd ever come back up ... if anybody remembered the cold start sequence. He got lucky, we only lost a day.
Admin
I work in a place where there is a BRB, right next to the door, that opens the door. So I guess I can understand why people might think the BRB opens the door. After years of "don't touch the BRB" it's sort of a WTF to actually use it.
At home, I became completely convinced that my PC had a bad power supply. That is until I correlated spontaneous resets and exuberant tail-wagging around the reset button. I have since constructed a "molly guard".
Admin
We just used a paperclip to cross pins 1 and 3, well 15 minutes later we made a switch but we used a normal SPST push button, which were much cheaper back then (and of course we had dozens of them at hand): http://www.mouser.com/search/ProductDetail.aspx?R=R13-502MA-05-BRvirtualkey11230000virtualkey112-R13-502MA-R%2fB
Admin
When we did an install of new UPS, after all the equipment was shut down, we drew lots to see who had the honor of finally hitting the BRB. Alas, it wasn't me.
BTW--- the hell with bring your kid to work day, I am lobbying for bring your DOG to work day.
Admin
I used to work at the same facility that aquatone282 mentioned above, and once a year we had to take the BRB apart, disconnect the Halon tanks from the system, and press the BRB to make sure that it really worked. I was so worried that I was going to miss a power cutoff or Halon connection that I triple checked everything before pushing the button. I'm glad to say that everything worked and there were not accidental power outages or Halon discharges. Then we put it back together and waited another year.
Admin
Look up "Molly Frob" in the jargon file.
Admin
I had a moment similar to this a few years back when I was at a web hosting company who's name I want say. We had about 20 servers, near Lucas Arts ILM render farm which at that time was in Exodus in downtown Los Angeles.
I had been planning to move the 20 servers and after I moved the last customer I called the datacenter and asked the person on duty to kill the power to our cage, and gave him our name. He said he would "call me back on his cell at the cage" for "security purposes"
When he called back he was yelling into the phone (noise datacenter) asking which cage was ours, I kept telling him.. Then I listened as it got really, really quiet. He was like. All done now. Then he asked me why Lucas would be shutting down all those servers that were just installed, and if I wanted them powered back up.
Needless to say, I almost DIED laughing and crying, I told him to turn the power back on, and made every attempt to call ILM, he said he would call.
I still tear up to this day thinking about the security guard calling ILM to let them know he accidentally turned off there 200+ server farm.
In case you are wondering, (I shouldn't be saying this) but if you know what datacenter a company is in, and the datacenter number, there probably a 95% chance you can call it WITH NO PASSWORD and get them to turn off or cycle the power.. VERY SCARY.
Admin
I do have to add one more comment. Back when "MCI Worldcom" was around, I toured their building in Columbus, Ohio. There main OC-192? Whatever connection to the west coast went through this small room. Normally you aren't even allowed to go back this far, but I was with my friend and he insisted on showing me the room. In this room, there were two guys sitting in Lab Coats, browsing on their laptops, just goofing off. The room was about the size of a large closet and had a small bulletproof glass window to see them through, it had double doors on either side. But inside that room, were two red phones, and a small center island table with two large switches. My friend told me that turning those switches off would complete turn off the internet and about 90% of all telephone service.
Impressive site to see. Apparently one of the phones went straight to a government call center so they could call in if they had reason to turn off the "net" or communications.
very strange.
Admin
WTF, no kids allowed. never mind how much the 3 milliseconds cost the company
Admin
I too have a MCI/Worldcom story. I was just hired as a contractor to work on some software and was given a cubicle to work in. I go to the cubicle and there is a desktop running with the default redhat screensaver.
"Cool" I thought, I was afraid I would have to develop on Windows NT 4.0, the thought of doing linux work on a linux desktop made me more comfortable with the whole project, which I already had lost confidence that it was a professional project and began to realize it was going to be a massive hack job.
So I move the mouse, username/password was the prompt, I look under the keyboard/mousepad/mouse, no username or password was given to me. "Weird", I thought, I'll just wait for the IT guy, no problem.
The IT guy shows up, with a new (as in new to me, it was actually 3 years old at the time) desktop, saying "Here is your desktop, oh hey, what's that computer for?". I say, "oh, I don't know, should I move it?". "Yup", was the answer.
So I can't remember how I turned it off, I might have just pulled the plug. I set up the new computer, boot into windows, get email set up, start downloading Vim etc.
Around an hour later, some guy runs up to me, says, "what happened to that computer?!?!?!?".
Me, "oh, its under the desk over there."
Him, "That was the production machine for the Texas Project!!!"
Admin
yeeeears ago, I worked with a development team, using NCR Towers (Unix) multiusers systems, the good old days of C, curses, text terminals, Unify databases. One day, a programming tech was hired, fresh from school. He had worked only with PC's (MS-DOS back then, Windows 1.1 was not introduced until one year later). So, he was going at best he could with Unix/C/Curses. One day, a program he was testing/debugging, crashed. The elder of us, will remember that if a Curses app. crashed, the terminal was usually left in "cooked" mode, so you were left with no echo, frozen cursor, etc. Being a DOS user, he knew that the solution for a frozen computer is Alt-Ctrl-Del right? ... but obviously it did not work. And if that did not work, what does a PC user does next? of course!! power off/power on, right? right! so, (all this without asking no help or advice from any of his mates sitting right next to him) he went to the computer room, and promptly pressed the power switch on a NCR Tower server, where 20 more users were connected. AND, the same elders will remember, the boot time for a NORMAL reboot on a Unix system back then was 15/20 minutes. After an abnormal shutdown, you have to add another 40 minutes for the fsck run. And the Unify database recovery. And the 20 irate coworkers whose work was lost (some were lucky with vi -r). The development servers were in a simple glass walled room, without all the fancy security controls used in the production datacenter. I think that since then, he was never again seen even in the same building with a power switch.
Admin
Admin
We once had a auditor (from our Blue Chip parent company's IT department - an organisation comprised almost entirely of managers) walk around our data centre, look closely at a one of our multi-CPU Sun boxen and poke the power button with a pen.
Everyone standing in the room stared at him with a What the F...? expression, whilst it slowly dawned on him what he'd done.
We never did tell him it was test server that wasn't being used...
Admin
It only takes once 'incident' for those buttons to get the plexiglass treatment.
In a facility I worked in, the incident was a network guy doing work above the ceiling panels, and backing into the button, while crawling out. The buttons were emergency power cut-offs for the entire server room.
Within a week every button was covered.
Admin
perhaps 9/11 WAS a coverup?
Admin
Bring-your-kids-to-work day at the White House:
Jenna: Daddy, what does this button do?
[click]
Admin
I think you mean yourself.
Admin
At a place I used to work many years ago, there was a BRS on the processor, and another on the wall beside the exit from the climate-controlled room. The one on the wall was for the air-conditioner. You know, in case of a runaway cooling event.
Admin
Parents often bring the kid(s) to work, which is distracting and dangerous -- productivity plummets because it's hard to concentrate on business when a toddler is rearranging your files, pouring your coffee into your keyboard, or screeching in the background, and adult workplaces are unsafe for little children because of sharp corners on furniture, doors that squish little fingers, heavy machines that can be pulled down on little heads, and stairs to fall down. And what toddler isn't curious to discover whether the yellow push pins on the bulletin board taste different from the red ones (and let's not forget about insurance liability!)
Jerry Steinberg Founding Non-Father of NO KIDDING! The international social club for childless and childfree couples and singles www.nokidding.net; [email protected]
Admin
[Stabs power button on Ultra10 next to desk]
Nope... power still on. Maybe I set up power management correctly...
"STOP-A power-off" did the trick though... but that's not likely to be something you type accidentally.
The power switch on the back would have been more likely to cause a problem, but that's a different story.
[Tries the other Ultra10's power button] That one worked... but it went through standard shutdown (judging from hard disk activity)
Admin
Admin
Oh the red button there... don't ever, EVER touch the red button.
(from Men In Black)
Admin
I have a "big button" incident to relate. It's not quite the same fuckup scale (it took less than a minute to reset, rather than 72 hours), but it still startled me. A few years ago I was visiting a major university here in Texas. I walked out of the parking garage through an entrance and noticed that there was some kind of sensor at the top of that metal arch over the entrance.
You know, the arch that shows the max clearance and has a bar on hangers to bump into your vehicle so you can know you're about to shave off the top of your big truck? Well, there was also an obvious sensor beam thingy up there at the top.
The temptation was too much, so I jumped up and waved my hand through the beam. Suddnely, a FSCKING LOUD alarm went off! See, it wasn't enough for them to have just a big metal bar to keep idiot truckers and RV'ers out of their parking garage, they wanted to be absolutely certain by making lots of noise and flashy lights.
I sheepishly wandered off, and after about 30 seconds the alarm turned off automatically.
Admin
How old is the CIO? 21?
Admin
The keyboard on my work pc broke and it was replaced with one that had these sleep/suspend/power buttons right below Del, End, and Page Down. What genius thought of this arrangement?!?! I use combinations of VPN and Citrix and SSH to connect to healthcare networks, and every time I missed the Del button I'd hit the stupid sleep button and would lose not only my connections but all work open in Citrix and ssh sessions . @(Q$(#. After a week I pried the keyboard apart and removed the 3 keys.
Admin
http://blogs.msdn.com/dmuscett/archive/2005/01/06/347523.aspx
Admin
i have three children ii took them to work with me and they were jumping every where i did not know what to do i worked at a phone shop and when i went to sell a phone my kids had locked thm thelves in the phone closet and then the manager came in and unlock the stock room to find that they had broken all of the phones i got fired
Admin
azor fiqyeugs sbdf xceqadzn arwviu zqvphn ujnogmhti
Admin
Not quite an IT Fuckup, but you'll get the idea...
Background: in some industrial environments with flammable materials, it is neither safe nor cost-effective to use Halon as a gaseous fire supressant agent. Therefore, some use C02...
I was working at just such a facility that happened to paint car bodies - the paint plants (there were 2) each had their own mix rooms and bulk storage areas (paint & solvent were stored in the bulk storage which was connect to the mix/pump rooms which then ran lines out to the actual paint booth areas).
Each of these bulk storage/mix rooms was protected by a 20,000 (yes - 20 THOUSAND) pound tank of liquid C02 outside of the plants. The idea being, should a fire break out and get out of control, the sudden dumping of all the C02 would serve several purposes - namely entinguishing the fire and dramatically lowering the temperature to prevent a BLEVE event from occuring (Wiki BLEVE if you're not familiar with the term - those of us who've fought fires are). Now you're familiar with the setup.
One day I'm making my rounds through the bulk storage/mix room areas and am watching this electrician do some maintenance work. Suddenly I see him reach for the BIG RED BUTTON...
All I can say is thank GOD for the delay. Otherwise, I'd be dead and so would he.
Admin
Back before they nerfed some of the calls in Second Life it was possible to launch people millions of kilometers in the air, so high that floating point rounding errors did the most AMAZING things to your avatar.
I made an object called "don't touch me". It was a little red button, not a big one, with about 128 scripts all doing this "launch into orbit" trick if you touched it. As a safety feature, it asked if you really meant to touch it, nothing happened accidentally, you had to decide that you were really going to touch it.
When you did that, you basically disappeared. Sometimes you'd see yourself bouncing around on the screen as the program tried to keep you in sight despite the rounding errors. You often had to log in again to get back to a sane state.
It's amazing how many people still just had to do it.
Admin
And if said electrician was ever heard from again, it was only after a VERRRRY long period of "questioning", I'm certain.
Admin
Years ago I worked in a computer lab with big red buttons placed every thirty feet or so along the wall. It got to the point that none of us really thought much about them.
Then came the day when the plumber turned off the four inch feeder pipe and proceeded to take a sledge hammer to the rusted section, without determining the direction of the waterflow... I opened the door on my way to a meeting to see a small wall of water rolling down the hall towards the lab, and the dozens of workstations we had sitting on the floor.
I slammed the door shut and spiked that red button while ordering all of my co-workers to start putting workstations on boxes chairs and the tables as fast as we could. With water pouring under the door we managed to get all of them up out of the way before the room flooded. We lost a few O'Reilly X-Windows books, but no hardware or people thanks to the red button to turned off the power to all of the power strips we also had on the floor.
Admin
When I saw this, I immediately thought of this story:
http://blogs.msdn.com/larryosterman/archive/2004/05/07/128004.aspx
Admin
I worked a ski lift for the past 6 months, it had plenty of emergency stop (and normal stop) switches on it, if I can recall correctly, the one I normally worked had, 6 emergency stops and 5 normal stops. Needless to say, they tend to get in the way when their is more than 1 person around and they were bumped quite a few times accidentally. Starting it back up took about 60 seconds though.
Admin
We've got a button in our data center. We refer to it as the Change of Career Button. Of course I work for a porn company, so there really isn't much turnout for bring your kid to work day.
Admin
yeah you think? Are you the only one stupid enough to have to have typos pointed out to him?
Admin
Often, due to being a flight commander, I was the officer in charge of a small Air Force facility. In fact, as the unit worked shifts to enable 24 hour ops, often I was the only officer on the unit. Sometimes, when I wasn't on scope, I got bored.
The main building dated back to just post WWII. This building had all sorts of interesting stuff attached to it over the years, the coolest bit being one of the big (but not humongous V8-powered) powered air raid sirens on the roof. It looked like a bench grinder about two metres long, and a bit less than a metre in diameter.
You can guess the next bit - yep, I couldn't help myself one night, so at around 2100 I flipped up the cover and hit the switch..... and was instantly disappointed, as buggerall happened.
Well, for a few seconds.
Then the decently large electric motor spooled up slowly - you could tell, as it was directly attached to the siren rotor/head/fans that were on each end, so it started out low pitched and not particularly loud. After about 30 seconds, it was enough to bloody your ears, and somewhat higher pitched - my test was complete, my nerve failed, and I hit the BRB again, expecting instant blessed aural relief.
Nope. The damn thing took nearly five minutes to wind down, giving the Base Commander, the Duty Officers of four surrounding Defence bases, the nearby hospital, three different lots of coppers and assorted others a chance to try and be first to inflict pain in my direction. My CO beat them to the punch, but they all got to kick my arse around for a day or two.
It was worth it - I didn't buy a beer for a month!
(name, rank and service number withheld...)
Admin
Our big red button fires CO2 into the data center. That would have taught the kid valuable lessons in both button-pressing and biology.
Admin
probably about 25 years ago @ School I did tour through Lucas Heights with my physics class.. Dont recall any such button but it wouldn't surprise me!
Admin
I worked in a large datacenter in the UK a few years ago. Next to the exits of the server room were the big red buttons of mass destruction.
Our datacenter manager was moving some new kit into the server room on a trolley, and as he opened the door he reached around inside to steady himself and slammed the big red button.
The whole building went very quiet very quick and emergency lights came on. Oh to have had a camera to capture his face when he did it!!!
Admin
I worked in a large datacenter in the UK a few years ago. Next to the exits of the server room were the big red buttons of mass destruction.
Our datacenter manager was moving some new kit into the server room on a trolley, and as he opened the door he reached around inside to steady himself and slammed the big red button.
The whole building went very quiet very quick and emergency lights came on. Oh to have had a camera to capture his face when he did it!!!
Admin
I worked in a large datacenter in the UK a few years ago. Next to the exits of the server room were the big red buttons of mass destruction.
Our datacenter manager was moving some new kit into the server room on a trolley, and as he opened the door he reached around inside to steady himself and slammed the big red button.
The whole building went very quiet very quick and emergency lights came on. Oh to have had a camera to capture his face when he did it!!!
Admin
Oh, for heavens sake, folks. You are not a real geek unless your kids have heard the mantra "No touching buttons, knobs or dials" from the moment they can first inch their way across the floor towards them.
Admin
I was a shiftwork operator running the biggest ICL Mainframe installation in the southern hemisphere (a dual-node 3980 and a 3950). We had a Big Red Stop button mounted quite high up on the wall near the airlock entrance. We'd run out of space to stock lineflow (continuous paper) boxes in the printer room so decided to put some boxes in the main machine-room. My shiftworking partner was doing a conan impression by carrying in three boxes stacked on top of each other (they are heavy!) and misbalanced managing to smack the Big Red Button. Darkness and silence except for faint ticking sounds as the machines cooled down.
Admin
Admin
I boought my son into work many years ago. I think he was about 3 or 4. I was in the PC room and turned my back for a minute. He had gone through the menus, just pressing enter and "Format the C drive" was the last one at the end of the chain!
After the disk was restored, the menu got changed!
Admin
Yeah, they have big red buttons on ventilators, CT scanners, MRI machines and anesthesia machines. The one on the MRI is particularly impressive when it vents all the liquid helium to the atmosphere...
Admin
I know the feeling. Try having a big red button that not only powers down the data center, but the entire campus. This was a DC designed in case of tornado, nuclear attack, or whatever. I got stuck in a remote office for 2 hours till they were able to bring everything back up.
Admin
My Big Red Button Story has nothing to do with kids at work. It has to do with stupid people putting two Big Red Buttons next to each other. It was a data center at one of our top military installations. The Big Red Button that turns on the flashing red lights to announce "Uncleared Personnel" was placed next to the Big Red Emergency Power Off Button. Sure enough, one day some one powered off the entire data center when they wanted to turn on the flashing lights. It only took me a day to recover my UNIX servers. But it was three days before the IBM 390 was back to normal.