• Tack (unregistered)

    This smells frist

  • FlayedJellyfish (unregistered)

    Have the XP operating system load from write-protected embedded solid state memory. It's the only way to be sure.

  • bvs23bkv33 (unregistered)

    truewtf is "unreal" facepalm

  • Uli (unregistered)

    This is why there is a Embedded Version of Windows XP featuring the "Enhanced Write Filter".

  • Mark (unregistered)

    “DO NOT HOLD THIS BUTTON TO TURN OFF THE MACHINE. DO NOT SHUT OFF THE GENERATOR WITHOUT FIRST TURNING THE SYSTEM OFF PROPERLY.”

    Typical developer not having the faintest clue about how to write clear instructions. The sentence above merely is some sort puzzle about what not to.

    Also why not create a new system restore point AFTER INSTALLING the whole presentation stuff?

  • David-T (unregistered)

    I know turning the power off without shutting down properly can cause issues occasionally, but generally you get away with it. Sounds like a load of bollocks.

  • Ron Fox (google)

    TRWTF is that Smell-O-Vision didn't overlap in time with Arduinos.

  • Andrew Miller (google)

    Install a small UPS, link it to the PC and set the PC to shut down when the UPS loses external power. Then they can go back to just pulling the generator plug.

  • Facepalm (unregistered)

    So this dude wasn't smart enough to create an image disc on the DVD? Serves him right.

  • Andrew (unregistered)

    TRWTF is that he picked up the phone on vacation.

  • Hannes (unregistered) in reply to Mark

    "Also why not create a new system restore point AFTER INSTALLING the whole presentation stuff?"

    That's what he did. But they corrupted the file system and killed the hard disk.

  • Blurb (unregistered) in reply to David-T

    Yeah, it will cause problems only occasionally when you cut the power occasionally. If you cut power every day, maybe more than once per day, all week long, your FS will be messed up rather sooner than later.

  • Church (unregistered) in reply to Uli

    "This is why there is a Embedded Version of Windows XP featuring the 'Enhanced Write Filter'."

    That was my first thought as well. This is what happens when companies stick to what they know, even if it isn't the right tool for the job. Whichever "Sales Engineer" didn't raise alarms when this sale was going through should be fired.

  • IrritableOldFart (unregistered)

    Sounds like an attempt to deploy standard office desktop equipment into the field without enough consideration of the environmental" factors. And then a series of over engineered unsolutions that just compound it.

    Not a good environment for a windows OS, speaking as a microsofty, I'd still have gone with Linux for that sort of thing (and have done in the past).
    Use a computer designed for field operations, even a battery powered fit-PC would've done it, then you can come up with a simple relay from the generator to set the thing shutting down.
    If that's too hard, stick a small intelligent UPS in the mix that can initiate a clean shutdown when it senses the power going.

    And yes, the instructions suck, but also, the mechanical solution to a button that only sends a pulse when pressed, no matter what they do to it, is trivial and cheap. Why do people always think they are the first person on the planet to encounter a problem and then go about knocking together a bizarre Heath Robinson machine to "fix it"?

  • Grzes (unregistered)

    Arduino is an overkill here. A capacitor, a resistor and a relay would do fine.

  • MuadDumb (unregistered)

    TRWTF is the guy custom brewing an Arduino pulse doohickey for the off button. That was your first thought? Get an APC battery backup instead. They have a USB port and a bit of software that you can configure to shutdown correctly in the event of power loss. And this works - I used to have a job putting windows pcs inside of industrial cabinets.

  • Decius (unregistered)

    Never underestimate the ability of military grunts to break their stuff.

    That said, I don't see why the setup described needs to write to a filesystem. Run it off of a bootable CD/DVD, and have a space on the mobile unit to store a local backup or three where it isn't going to get physically damaged and make sure that the MSGT also has a few backup discs. Discs are cheap, downtime is expensive.

  • Rahan (unregistered)

    Couldn't they just go in the bios and disable the shutdown on hold feature? I've done that.

  • Decius (unregistered)

    Case in point: I have no idea what happens if the custom power button hack receives a morse code impression of a creeper (SSSSSS), or ditditditditditditditditditditditditditditditditditdit at 100 WPM. But if it makes the monitor go black faster (shuts down the system), then that's exactly what the grunts will do every gorram time.

  • MaxArt (unregistered)

    An Arduino, really? There are common buttons that just send a pulse even if kept pressed., at the cost of what, 50 cents?

  • I dunno LOL ¯\(°_o)/¯ (unregistered)

    I once worked with a system that was made for trainee testing in the USAF. The computers were strange 68K-based systems running a Unix clone (called Regulus), built into a VT-100 clone terminal, with a PDP-11 bus (and therefore data written through the SCSI controller had the NUXI problem). They were built into steel boxes that opened up into testing booth stations and were shipped via cargo transport.

    This was also shortly before hard drives became self-parking. (that happened at 40MB, these used 10MB drives, and were getting 40MB drives as part of the phase I was working on)

    So not only was it a Unix clone with a write-cached file system, but the often un-parked hard drives were shipped via military cargo planes. Not surprisingly, they had a lot of problems with failing units. It also had a cartridge tape drive that it could boot off of, but the tape boot method was slow enough that the tape was constantly rewinding every few seconds to go back and catch up with the missed blocks.

    Oh, and when it booted up, it originally started into a pre-Macintosh version of MacsBug, and you did a G command to a specific boot loader customized for each type of boot method. I was at least able to add code to the boot ROM to put in an actual startup menu, with an actual item for "park hard drive". Not that it actually got used, but it was better than having to remember something like G FF0380.

    I still regret not saving a copy of that boot ROM image with its primordial version of MacsBug.

  • (nodebb) in reply to I dunno LOL ¯\(°_o)/¯

    "This was also shortly before hard drives became self-parking."

    Tsk. Control Data SMDs were self-parking long before PCs even had hard disks. 'Course, they had to be, since they were removable pack drives where the heads were part of the drive, not part of the pack, but you know what I mean.

    And for real fun, consider the poor drive my dad and I saw when I worked at his company one summer. It was a 75 MB SMD that, therefore, took five-platter packs. On this occasion, the electromagnetic brake on the drive spindle failed. And of course, it failed "off" so that when you asked the drive to unload the pack, the pack didn't stop spinning.

    Why not? Well, it was five platters(1) each 14 inches in diameter, in 3mm aluminium. That has ... significant ... angular momentum when it's spinning at 3000 rpm, and, more to the point, it also has significant moment of inertia, and doesn't slow down remotely rapidly without assistance. We did eventually get it stopped, and my dad changed the brake doohickey, and then it worked correctly.

    (1) One guard platter, three data platters, then one guard platter. The data platters, between them, had five data surfaces and one "servo" surface that helped the heads stay aligned.

    Addendum 2017-05-18 10:37: The guard platters were there to protect the data platters from mechanical damage.

  • Nobody (unregistered)

    TRWTF is that they should have been running windows embedded or linux from a read only partition.

  • foxyshadis (unregistered)

    Yeah, TRWTF is definitely submitter and their company, who went through months of twice-weekly calls of the exact same technical problem by trying to berate the ever-changing untrained users into working around it, before finally coming up with an overengineered fix that still won't work if power is cut. You're definitely not the first to have such a common problem.

    The time to start figuring out a permanent solution is as soon as you fix it the first time, if "will this happen again?" is very likely.

    Unless he just enjoyed getting out of the office for a few hours on those days, I suppose.

  • Sole Purpose of VIsit (unregistered) in reply to foxyshadis

    You do sort of wonder, don't you? I mean, it's the Law of Three made incarnate.

    1. We send our product out to remote sites with simple instructions. This turns out to be expensive, because we have to send a techie out twice to fix a ridiculously trivial problem. Third time round, we get somebody to write a sticky with strict instructions. (Although, this being the military, I think writing them down for the commanding officer, or better still the drill sergeant, would have made more sense.)

    2. Nobody pays any attention to those instructions. Twice. OK, time for a hardware solution. After all, this is effectively an embedded XP console on the back of a truck. Let's admit that we got it wrong (and let's face it, there is no point at all in blaming the grunts who are moved in and out every month or so), and install a UPS with a trigger.

    3. It's just gone wrong again, twice over. We need a better solution here.

    I see TRWTF as the management here. But then again, 99% of the time, I always do.

  • Herby (unregistered)

    Moral of the story: Read-only partitions are your friend. Make the CD/DVD bootable and have it do an automatic re-image as its task.

    Yes, users are GRUNTS. Deal with it.

  • Sole Purpose of VIsit (unregistered)

    Ah, yes. And I think it's worth pointing out that I don't blame Ron at any point here. It's tempting for all of us to do so. We all know that we would have diagnosed the problem better and faster and written up the solution immediately. Basically, this is Confirmation Bias. We have all been there.

    I see the problem as a totally fucked up value system in IT middle management. Nobody in middle management wants an ultimate fix to a problem, because ... YAGNI.

    Instead, these pin-head morons devote themselves to "metrics" and "cost-benefit analysis" (which never is) and bean counting. Which is how you come to the position where you have just spent $10,000 on a couple of flights (etc) to West Texas, and you still[/b} haven't solved the problem, despite your best efforts.

    [b]Output, not [b}outcomes[/b], I'm tellin' ya. Nobody in middle management has [b}ever[/b] been able to measure a "problem solved."

    Because, by definition, you will never see that problem again.

  • Sole Purpose of VIsit (unregistered)

    (And back to the early 90s. No edit button, I see.

    (Which would be OK if we had a RTF sort of form entry system, but basically sucks if we can't guess ahead of time what Mark-Thing is gonna do.)

  • (nodebb) in reply to Sole Purpose of VIsit

    if we can't guess ahead of time what Mark-Thing is gonna do

    Proof-reading would have shown the problem* just fine, I’d hazard a guess.

    • Pressing Shift when you shouldn’t have.
  • yup (unregistered) in reply to Andrew Miller

    my thoughts exactly

  • Krzysztof Słychań (google) in reply to Grzes

    Not even a relay needed. Standard mobos use a NO power button, probably puled up to 5V SB; the input is negative logic. You just need a simple differentiating circuit to make the input change its state back after a second or two. All it takes is a capacitor and a parallel discharging resistor; these two wired in series with the button.

  • Krzysztof Słychań (google)

    Finally, some PEBKAC instead of badly written code or poor management...

  • Gus (unregistered)

    Nice try, but as others have mentioned, an Arduino is about 95 million percent overkill. A 47uF capacitor and a 1 megohm resistor in parallel will do the trick. Or you can write explicit instructions, NOT telling them what NOT to do, but telling them exactly what to do, along the path of:

    "SHUT DOWN SMELL-O-VISION BY PRESSING THIS BUTTON FOR ONE TO TWO SECONDS. THEN WAIT FOR THE DISPLAY TO GO COMPLETELY DARK. THEN SHUT DOWN THE GENERATOR. NOT FOLLOWING THESE INSTRUCTIONS IS A STOCKADE OFFENSE".

  • Foo AKA Fooo (unregistered) in reply to foxyshadis

    "The time to start figuring out a permanent solution is as soon as you fix it the first time, if "will this happen again?" is very likely." -- Also, if it happens to any customer, it can happen to all of them (including your boss).

    "Unless he just enjoyed getting out of the office for a few hours on those days, I suppose." -- Indeed, I think he was actually collecting miles.

    WRT other comments:

    "read-only" -- yes, or at least a journaled FS and/or synchronous mounts should have reduced the damage significantly

    "UPS" -- sure, or a laptop set to shut down or hibernate on A/C loss, basically using its battery as a UPS

    "capacitor, resistor, etc. instead of Arduino" -- yes, or a button connected as a pseudo-keyboard or so, initiating a proper shutdown

    "better worded instructions" -- NO! Human solutions are always inferior to easy technical fixes.

    "downloading all the software over his phone’s mobile hotspot to his laptop" -- WTF? He traveled to a remote location and didn't even have a copy of their software with him (on DVD and/or USB, encrypted if worried about confidentiality in transit)!?

    So, a lot of WTFs here, and none on the customer's side. I've never worked for the military, but even I know a thing or two about building robust systems. At least the military know nows which company never to consider for any serious systems.

  • positivejam (unregistered)

    I couldn't read this without a mental twitch every time the author referred to guys, who were almost certainly pogues, as grunts.

  • yep (unregistered) in reply to Andrew Miller

    yeah, that's the obvious answer

  • Quite (unregistered) in reply to IrritableOldFart

    THIS.

    TRWTF is using a Windows PC.

    I remember a delightful system I used to work on way back in the dawn of history, when computers were real computers whose bootstrap routine you knew by heart and could recite from memory. There was a power-off system which had a hefty inbuilt capacitor whose job it was to maintain power for 2 long and lovely seconds before powering down the system, during which time the emergency routine was invoked which stored the immediately relevant state space, so it could be started up from where it left off when the power was returned. (This was in addition to the battery backup.)

    I kind of miss those reassuringly solid and failsafe military-grade machines.

  • Quite (unregistered) in reply to positivejam

    By "pogues", don't you mean "pongos"?

  • Paul Neumann (unregistered)

    So, software that operates pumps and fans based on timed events ... Why is it writing to system files so much that most random power events cause a filesystem failure? Why is it writing to system files at all? Why is it even writing any files?

    Seems to me the Arduino would have been a great replacement for the whole system, rather than the power button!

  • gnasher729 (unregistered) in reply to Decius

    Military are well known for being able to follow precise instructions exactly. The instructions they were given were rubbish. Just "Press the power button for exactly half a second. Then wait exactly three minutes. Then turn the generator off".

  • CrazyEyes (unregistered) in reply to Mark

    It's amazing to me that developers don't apply proper code-writing principles to their own speech. "DO NOT..." is not how you tell someone to DO something. Much in the same way it's considered improper to test for a "not" condition in an if statement, instead test for the condition you actually want to happen. "PLEASE PRESS THE POWER BUTTON BRIEFLY; DO NOT HOLD" would have been clearer. Of course it's hard to nitpick over things like this when you don't know how many details get lost in translation. The real punchline is definitely his own boss not understanding this after all this trouble.

  • Adzy (unregistered)

    Probably a bit late, but I'm calling horsefeathers on this one.

    I'm surprised that they had this problem at all even with16 year old WinXP. For starters, NTFS is journaled, and in 20 years in this industry have almost never seen an NTFS based OS become unbootable due to improper shutdown.

    During that particular era, I worked as a software developer for a company that made entertainment lighting controllers. Of course we had our super-high-end version based on ruggedized embedded hardware with triple redundant everything, but our little mickey mouse model was essentially just a small WinXP machine with a USB to DMX converter.

    Those things got abused quite a bit and were almost never shutdown properly. We shipped the units with drives imaged with our software pre-installed, system restore (and just about everything else) disabled and had no such problems. Unless the OS is constantly overwriting itself during normal operation (which it doesn't), there is absolutely no reason for it to become corrupt due to a failed shutdown, same with your app or "presentation" file.

    Occasionally chkdsk would eat a user's "show" file, but that was rare as hell, since we wrote a new file each save to allow undo to work even after a reboot, and any user who cared backed their shows up on floppy at the time anyway.

  • IrritableOldFart (unregistered) in reply to CrazyEyes

    Of course we've not seen their actual code, I do regularly stumble on stuff like;

    If not(No_Value_Provided_Yet=false) then 'Nothing to see here! Else ''Some painful inverted logic End if

  • Jeff Go (google)

    "Then Ron spent the day re-installing and configuring their system by downloading all the software over his phone’s mobile hotspot to his laptop, then transferring it via USB stick."

    Really, this is a wtf as well. This should have been in his "toolbelt" before he left the office.

    And, for all the money they spent, they could have easily purchased a UPS brick which would automatically shutdown the system and allow it enough power to complete the process. Then the grunts can hit the mains button and be done with it.

    But, no, they had to go through an arduino solution... which seems like one more point of potential failure.

  • Gene Wirchenko (unregistered)

    The people who are suggesting an automated system to handle the shutdown are forgetting the human factor. It is quite possible for someone to say, "The system was not shutting down, so I _____." Fill in the blank with such goodies as "turned off the generator", "pulled the plug", etc.

    After all, they were ignoring a note that told them what to do. What makes you think they would follow a procedure that they thought did not work?

  • eric bloedow (unregistered)

    this reminds me of a story i read somewhere: a woman complained that her computer was taking a long time to start up in the morning.when they investigated, they found that she had the ridiculous habit of pressing the power button TWICE when leaving for the day and TWICE MORE after coming to work, causing TWO improper shutdown problems EVERY day...

  • tlhonmey (unregistered)

    To understand why it was crashing so often, you have to understand the mindset of the average non-technical user.

    They yank the power or hold down the button to turn it off. This by itself has a chance of corrupting the system, but really only a fairly small one as the file system check on the next startup will fix it.

    However: On next startup they notice that it's not coming up as quickly as they'd like, and they're on a schedule dangit! So they frob the power switch repeatedly until it either breaks or they get a chance to skip the check. Most of the time it will break.

  • SomeName (unregistered) in reply to Adzy

    Even later reply, but I believe this story. I used to have my own 800 MHz computer with 30 GB drive and it would happily corrupt OS during improper shutdowns. For some reason it also happened only on Win XP, but never on Win 98.

Leave a comment on “The Smell-O-Vision”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article