• (cs)

    Monrobot?

    [image]
  • customer service my .... (unregistered)

    Here is a bit of irony, at least in my mind.

    I just got off the phone trying to call my insurance company. I wandered for 2 or 3 minutes through their automated phone system to get to a customer service rep who could do nothing but tell me the "system" was down. I came back to my desk, thinking, it shouldn't be too hard for a monitoring system to send a message to the phone system, having the automated attendant alert soon to be angry customers that they are experiencing technical difficulties, and to try back later.

    I immediately came to this site for a laugh to cheer me up, and it worked. Thanks, the daily WTF, you are my hero.

  • Ed (unregistered) in reply to Domo Arigato

    No budget, it seems.

  • Carl (unregistered)

    That's either gotta be a very sensitive power button or a powerful CD drive for it to push it hard enough to actually trigger the button.

  • (cs) in reply to Josh
    Josh:
    Did anyone else read the title as IT Tampon Robot? I thought this story was going to take a HUGE twist.

    Damn dyslexia.

    I could not decide if it was: IT-APP-MON-ROBOT or I-TAP-M-ON-ROBOT or (with a Jamaican accent) I-TAP-MON-ROBOT

  • OhWow (unregistered)

    THAT IS AWESOME! That Man Deserves a saulte and if I ever ran into him I'd buy him a beer! Yes, it could have been a better situation, but when it comes down to it, sometimes you've gotta do, what you've gotta do, any way you can!!

  • mark (unregistered)

    That's a great story!

  • sk (unregistered)

    oh, such a sad story...

  • (cs) in reply to Domo Arigato
    Domo Arigato:
    Wouldn't a UPS controlled by the 2nd machine have beeen easier? Just cycle it when the machine stopped responding.
    That assumes they HAD a spare UPS. Budget freeze, remember. Also - what might the result be of placing 2 CD drives such that when they eject, they push each others eject buttons B-)
  • Neon Knightmare (unregistered) in reply to Morty
    Morty:
    MacGyver:
    Awww, come-on guys, this is not a WTF! This is classic seat of the pants make the best of a bad situation with some techie innovation.

    Amen. The situation is a WTF, but the solution is a brilliant hack, and not in the Paula Bean sense. He reused old hardware to create a workaround. It was better than what they had before. No budget was spent. Brilliant!

    Don't you mean "Brillant?" ducks

  • (cs) in reply to Erik

    If theyu'd plugged in one little cable the poor old ITAPPMON Robot could have spent it's twilight years thinking it was still performing a vital task. Instead, it was consigned to a life of futility, with chronic pain around the CD-tray. Oh, the inhumanity, to just push it aside without even a gold watch for its years of faithful service.

    And that's why computers hate us.

  • asdf (unregistered) in reply to Zygo

    wait, so it'd potentially reset a perfectly working machine simply because it hadn't seen disk activity in 4 minutes?

  • (cs) in reply to Izzy
    Izzy:
    If theyu'd plugged in one little cable the poor old ITAPPMON Robot could have spent it's twilight years thinking it was still performing a vital task. Instead, it was consigned to a life of futility, with chronic pain around the CD-tray. Oh, the inhumanity, to just push it aside without even a gold watch for its years of faithful service.

    And that's why computers hate us.

    Izzy, run thee spill chucker, dammit.

  • dust (unregistered) in reply to Domo Arigato

    budget freeze.

  • Aaron (unregistered)

    The only amount of time that machine death was shortened by was the amount of time it takes to drive to the datacenter and reboot the machine manually when it dies in the middle of the night.

    I'm a controls engineer, so I'm looking at this from outside the IT/software bubble.

    Starting and stopping equipment is usually where a failure occurs, because those are the most physically demanding times for the hardware.

    Rebooting a server is hard on the equipment. The fans on the servers I'm using right now run full blast at startup, for example. You can hear these things rebooting from the next room.

    I'm surprised that the hardware lasted until the budget problem was resolved. That's all I'm sayin'.

  • Ivan (unregistered)

    oowww... what made me cry!

  • Pitabred (unregistered) in reply to Carl
    Carl:
    That's either gotta be a very sensitive power button or a powerful CD drive for it to push it hard enough to actually trigger the button.

    I take it you have never pressed the reset button on a desktop or server system? They take almost no pressure, but are typically recessed to prevent accidental contact.

  • mojonixon (unregistered) in reply to Morty

    "No budget was spent." If you add up the time in the meeting and the time putting it together, the opportunity cost is probably close to a cheap but acceptable replacement server. Guess I shouldnt be surprised this is so common in a society where "saving money" means spending money for crap on sale.

  • Steve (unregistered) in reply to Morty

    A friend of mine had a better version of this...he had a robot arm on a UNIX box connected to the power switch on a windows server. He could cycle the power remotely whenever he wanted.

  • (cs) in reply to BobPaul
    BobPaul:
    Most UPSs don't give you that kind of control. ... You can't normally say, "You know what? Kill the power to all connected devices" because manufacturers assume one of the connected devices is the one issuing the command.
    It's much more fun to play with enterprisey UPS systems... We had a combination Battery-Bank/UPS in conjunction with managed power bars that would let me remotely power down individual plugs on the power strip as well as shift power from utility to battery & back and then cycle the whole unit completely (same with our A/C unit; that's all KINDS of fun), and the whole system had enough juice to power 40 servers, 12 switches/routers, 180 POE devices, a 5ton A/C unit and about a dozen workstations for 8 hours (if the generator failed to kick in)... good times, good times

    If you're interested, have a look at Emerson Network Power's site (the company we got our gear from). Oh, and if you want a battery bank that can handle that load for that duration, you'll need to have a supporting structure that can withstand a 5ton load over 8 sq feet and a supporting surface that can withstand pressures of ~4000psi ;) Fun!

  • (cs) in reply to yet another Matt
    yet another Matt:
    I loved that story. I feel sorry for the poor machine in the corner tirelessly doing it's job. I want to give it a nice home.
    I had the same reaction! I was all set to come in and post "anyone else feel sorry for ITAPPMONROBOT?" I mean, "It spent the last weeks of its life dutifully opening and closing its CD ROM drive every two minutes, reaching in vain for the restart button that it'd never touch again" is just so poignant!
    Dave:
    Many of you feel sorry for this server. That is because you are crazy! It has no feelings, and the new one is much better.

    http://youtube.com/watch?v=yx43NJVYBg8

    Ah hahahaha!
  • Gedoon (unregistered)

    Oh sweet! This is the best WTF of all times! It's got everything: A Robot, McGyver-wannabe, half-assed mcgyver-solution for a hardware problem, sad ending for the robot, ping script, and a girl named Laura! I'll stop reading this blog now, 'cos NOW I'VE SEEN EVERYTHING! Thank you The Daily What-ever-you're-called-today for pimping my day!

  • Chris (unregistered)

    That is beautiful story, a WTF that incorporates hardware and software...the best part is ITAPPMONROBOT being plugged back in after it's server was replaced, and it's still sitting there vainly trying to do it's one job, the one job it was built for.

    There should be a video of that on YouTube.

  • Dave Greiman (unregistered)

    Now that's how you work on a tight budget.

  • dkf (unregistered) in reply to mojonixon
    mojonixon:
    "No budget was spent." If you add up the time in the meeting and the time putting it together, the opportunity cost is probably close to a cheap but acceptable replacement server. Guess I shouldnt be surprised this is so common in a society where "saving money" means spending money for crap on sale.
    You see it a lot in organizations where pay comes from a different budget than equipment, and that's most large organizations to be frank.
  • (cs) in reply to EvanED
    EvanED:
    yet another Matt:
    I loved that story. I feel sorry for the poor machine in the corner tirelessly doing it's job. I want to give it a nice home.
    I had the same reaction! I was all set to come in and post "anyone else feel sorry for ITAPPMONROBOT?" I mean, "It spent the last weeks of its life dutifully opening and closing its CD ROM drive every two minutes, reaching in vain for the restart button that it'd never touch again" is just so poignant!
    Nobody knew how to stop the script, huh?
  • Jake (unregistered)

    This reminds me of a story I heard once -- it was years ago and a FOAF thing, so I may not have it quite right, but it's another ingenious use of removable media:

    The originator of this story apparently worked for a company dealing in custom servers with network boot, network OS install, etc. -- basically obviating need for physical removable media. But they did custom builds, so when a server farm asked them for units with CD-ROM drives, they were accommodating but curious.

    Their client explained to them that they had an established procedure for rapid physical location of individual machines -- if they needed to find a particular server, they'd instruct it to eject the CD drive, then jog along the stacks looking for the machine with its tongue hanging out.

  • (cs) in reply to Tim B
    Tim B:
    I haven't felt this bad for a robot since I saw A.I.!

    A.I. only made me feel bad for Stephen Spielberg. WTF was that crap at the end about being able to clone people's memories, but they die after a day. That's the stupidest thing I've ever heard.

    / Surprised how many people thought those things at the end were aliens, not advanced robots though. I have lots of stupid friends, I guess.

  • programmer x (unregistered)

    This reminds me of a story from an ex-IBM employee who told me a story about one of their old mainframes which didn't quite work right. Every now and then it would just stop working for whatever reason and had to be reset, losing lots of impoertant information.

    This happened fairly often and they decided to rebuild it from the ground up to fix whatever was wrong with it. No luck: it still stopped working quite regularly, so they posted a guy near it at all times so that when it shuts down he just has to hit the button and it'd restart. Eventually, the guy got so pissed off at the machine he just kicked it; somehow it started up again with no lost information at all.

    The next day he shuffled over to my teacher and said, "last night when I it stopped I just kicked it, and it just started working again."

    My teacher looked at him and asked, "where did you kick it?" So the guy points it out and my teach got out a peice of chalk and put a circle where he kicked the machine. "Next time it breaks, do that again."

    So, eventually it broke again and the guy kicked it. The machine whirred back to life the way it did last time. After a week or two of doing this without losing any information they decided to report it to a senior manager.

    His response? "Well, add it to the manual."

  • Alex (unregistered) in reply to Domo Arigato

    The article specifically states "Great. And we can't replace the unit while we're in a budget freeze..." They didn't have any money to buy an expensive UPS unit.

    A rack-mounted power-bar would also do the same thing, but that costs money as well.

    The point of the story is that someone did something ingenious and cost-free in order to solve a problem.

  • TopCod3r (unregistered) in reply to operagost

    Can you e-mail me teh codez for thet app mon script?

  • AmericanTurtleCo (unregistered)

    I don't get it... this is a fictional story? If not, why the fake (Office Space) company name?

  • (cs) in reply to mojonixon
    mojonixon:
    "No budget was spent." If you add up the time in the meeting and the time putting it together, the opportunity cost is probably close to a cheap but acceptable replacement server. Guess I shouldnt be surprised this is so common in a society where "saving money" means spending money for crap on sale.

    But to management, the meeting and assembly time is invisible and therefore zero. They were already going to spend that money whether they had the meeting or implemented his solution or not.

    I'm not saying you're "wrong", I'm just saying that the PHB didn't see it that way...

  • me (unregistered) in reply to Josh
    Josh:
    Did anyone else read the title as IT Tampon Robot? I thought this story was going to take a HUGE twist.

    Damn dyslexia.

    Yes

  • (cs) in reply to AmericanTurtleCo
    AmericanTurtleCo:
    I don't get it... this is a fictional story? If not, why the fake (Office Space) company name?

    Alex always does this to make the stories anonymous. Welcome to the site. :)

  • AmericanTurtleCo (unregistered) in reply to GalacticCowboy

    Awesome. Then it really was a good story :)

  • arosebyanyothername (unregistered) in reply to Zygo
    Zygo:
    Mind you all my critical servers are set up this way (a ring o' UPSes, where machine N's USB port controls machine ((N+1) mod M)'s UPS)--although the monitor script will refuse to power-cycle a given machine more than once every 12 hours. If it doesn't come back up the first try, I want someone to check out the hardware just in case magic smoke is leaking out of something.

    I find that Nagios for system monitoring and network managed PDUs are a much more effective and scalable solution.

  • Vartan Christopher Simonian (unregistered) in reply to Josh
    Josh:
    Did anyone else read the title as IT Tampon Robot? I thought this story was going to take a HUGE twist.

    Yes.

  • Johnny Syntax (unregistered)

    This (nearly) exactly what we do to keep Microsoft IIS websites alive, except for the hardware. You see IIS has a big flaw, it leaks memory badly, and it is exasperated when the ASP code uses V.B. (Virtual Bs.) as its internal language. The monitor (script), often called a Nanny, attempts to open web-pages on the IIS web, and, when the web site is too slow or does not respond, it sends a re-start IIS command through a back-door. Restarting IIS releases the long-lost memory, and re-establishes all connections to the database and wha-la! things are up and running again. For busy sites a restart automatically happens about 1-2 times per 24 hour period. Fortunately the restarts are usually not noticed by the clients. They often take it as an internet failure but nope: its another GPF (Gates -Provided Fault)! Just use Java and get over it! ;) JS.

  • Rob (unregistered) in reply to Guesser
    Guesser:
    a much less crude solution would just be a relay or a transistor switch across the reset button leads, stuck on the other machine's serial port...

    A PC uses too damn much power to be given such a minor task. Totally. I once had to power-cycle a Toshiba cable modem far too often so I cut up a printer cable and used a ping script in the firewall with parapin to open the NC pin on a 5V relay stolen from an old modem. It could have closed a reset switch on a motherboard but I hooked it to a DC jack harvested from a worthless D-Link router and a chopped DC cord that both matched the Toshiba AC adapter, making a simple remote-controlled disconnect. It went into service but didn't get much action since the Cable Guy came to replace our modem soon after... and soon after that I helped the staff decide that it was time for DSL.

    The moral of the story: Solder fumes are dangerous.

  • (cs) in reply to brazzy

    Dude, that's the only Furutama episode I can't watch. I always skip it when I'm doing a DVD marathon.

    cries quietly into coffee

  • (cs)

    Oh come on the solution is so easy: Buy a sTEVE remote administration card (name changed) for a few hundred bucks so you can do a hardware reset from a remote location whenever it needs to be rebooted. The real WTF is that would probably fit into some budget. :P

  • (cs)

    Did the important server have ITAPPMONROBOT's back? No ping and it rebooted the robot? I wonder if a "ping" was the only thing the script checked. If IPAPPMONROBOT lost network connectivity, it would repeatedly pummel the production server.

  • (cs) in reply to Johnny Syntax
    Johnny Syntax:
    ...and it is exasperated when the ASP code uses V.B. (Virtual Bs.) as its internal language.
    You're using ASP and VB? Wow, how 1997! Is that the age of your IIS, too?
  • iToad (unregistered)

    No duct tape, bailing wire, or XML?

  • IT_NIK (unregistered) in reply to Zygo

    I call shenanigans. You thought about that circuit, but you didn't really build it.

  • SysKoll (unregistered) in reply to programmer x
    programmer x:
    This reminds me of a story from an ex-IBM employee who told me a story about one of their old mainframes which didn't quite work right. Every now and then it would just stop working for whatever reason and had to be reset, losing lots of impoertant information.

    This happened fairly often and they decided to rebuild it from the ground up to fix whatever was wrong with it. No luck: it still stopped working quite regularly, so they posted a guy near it at all times so that when it shuts down he just has to hit the button and it'd restart. Eventually, the guy got so pissed off at the machine he just kicked it; somehow it started up again with no lost information at all.

    The next day he shuffled over to my teacher and said, "last night when I it stopped I just kicked it, and it just started working again."

    My teacher looked at him and asked, "where did you kick it?" So the guy points it out and my teach got out a peice of chalk and put a circle where he kicked the machine. "Next time it breaks, do that again."

    So, eventually it broke again and the guy kicked it. The machine whirred back to life the way it did last time. After a week or two of doing this without losing any information they decided to report it to a senior manager.

    His response? "Well, add it to the manual."

    Great joke! I seriously doubt it's true, though. Neither IBM nor their customers would take repeated system crashes lightly. Heck, if you want everyone in an IBM mainframe support dept to take your issue seriously, you just tell the guys that it involves data loss. That will put you at the top of the queue right away.

    Once, I was in a room where the president of a bank explained that he was quite miffed because a DOS-based IBM ATM machine would just crash and reboot periodically. The IBM sales engineer said it wasn't their code's fault. The bank president said "I think you could find how to fix it and I'm just going to stop paying for my mainframe rental until you fix it." Biiig silence. The project was started immediately. I know, 'cuz I ended up coding the fix.

  • Miles Thompson (unregistered) in reply to Morty

    Not the first time I've heard of this either. I've heard description of the exact same solution being put in place at a datacenter in Wellington.. (not somewhere I worked but not just a myth about some random place either - i heard it direct from someone who set it up).

    Creative no 8 wire solutions, sometimes looking stupid is well worth it.

  • Miles Thompson (unregistered) in reply to Carl
    Carl:
    That's either gotta be a very sensitive power button or a powerful CD drive for it to push it hard enough to actually trigger the button.

    In the solution I heard they jerry rigged some kind of pen so that as the drive opened the pen pushed the button.

  • Charlie Reddington (unregistered)

    You should submit this to hackaday.com

Leave a comment on “ITAPPMONROBOT”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article