• (disco)

    Well, at least the devs won't know they exceeded the limit, because they don't need to know that pushing the volume up to 11 (and higher) doesn't actually do anything, amirite?

  • (disco) in reply to Tsaukpaetra

    Well, the fuel throttle never goes over 10, does it?

    Can't see what could go wrong when implementing some acceleration logic.

  • (disco)

    In this case the question is whether the calling code has the responsibility to supply safe positions only. It looks like it has that responsibility, so weakening the assert is of course the wrong thing to do.

    In production I often find myself first softening errors to warnings then removing the warnings altogether. But I don't work with life-threatening appliances either. The defaults are always safe as far as the user's physical integrity is concerned :smile:

  • (disco)

    Hi there, devil's advocate here.

    No, okay, so we weren't in the servomechanism business, but perhaps we ought to have been. This was 1993, and I was working abroad as a field software service engineer.

    See, we'd provided to the customer a homegrown screen editor. We had a cursor that could be moved up and down, and left to right, but not off the screen. So, what we did was, when the user pressed, for example, up-arrow when he was already at the top of the screen, the teleterminal beeped at him and wrote an error message to the screen saying something like "Attempt to move the cursor out of bounds."

    Now, the cursor took a little time to respond to a keypress, such was the nature of the female canine we had provided. So it was natural for the user, rather than press (wait) press (wait) press (wait).... to get to where he wanted to get to (the top of the screen, say), the natural response was to hold the up-arrow key down until the top had been reached.

    We knew when someone in the office had done this by the insane "beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep-beep ..."

    My first assignment was to change the code to "if cursor_vertical_position < 0 cursor_vertical_position = 0" and so on.

    Except that I hadn't actually been provided any of the source code, so I had to write a fax to head office and wait for them to deliver a new version of the software. That happened 18 months into my contract and there were so many other things wrong with it the customer quite understandably said they'd revert to the old version before it had been up 3 days. 6 months later I was sent home. I wasn't dismayed.

  • (disco)

    Michele S. had recently worked on code for an engine-controlled device. Since the device had physically-moving parts operated by servos, the software had to be careful to not move anything out of bounds because it could actually break a physical piece of equipment.

    Right, so they were building a Therac-25, I take it.

    Seriously, never, ever rely on software to protect hardware. Software has bugs, and if the hardware doesn't have protective elements in it, the bugs can damage the hardware, or even kill people.

    If you don't believe me about the "kill people" part, read about the Therac-25.

    Here's a link for you: https://en.wikipedia.org/wiki/Therac-25

  • (disco) in reply to JBert
    JBert:
    Well, the fuel throttle never goes over 10, does it?

    Mine goes to 11.

  • (disco) in reply to Steve_The_Cynic

    So in an engine control system, how do you mechanically prevent the system from letting the engine overspeed by applying too much throttle with too light a load? Bear in mind this will have to work correctly with everything from no load at all to a heavy overload on the engine.

  • (disco) in reply to RFoxmich
    RFoxmich:
    Mine goes to 11.

    Mine's in imperial units, so it goes to 12! :trolleybus:

  • (disco) in reply to gordonjcp
    gordonjcp:
    So in an engine control system, how do you mechanically prevent the system from letting the engine overspeed by applying too much throttle with too light a load? Bear in mind this will have to work correctly with everything from no load at all to a heavy overload on the engine.
    The word you are looking for is "governor" and these can be done in hardware. In fact, on 19th Century steam engines, they had to be done in hardware, because there was no software.

    EDIT: My point here is that the no-software overspeed protection problem was solved in the 19th Century.

    EDIT(2): Actually, I'm wrong. 17th Century, for wind-powered mills. https://en.wikipedia.org/wiki/Governor_%28device%29

  • (disco) in reply to Steve_The_Cynic

    So how do you get that to reliably control the speed of the engine, where the fuelling is done by a bunch of solenoids under software control and the throttle position is set by a stepper motor under software control?

  • (disco) in reply to gordonjcp
    gordonjcp:
    So how do you get that to reliably control the speed of the engine, where the fuelling is done by a bunch of solenoids under software control and the throttle position is set by a stepper motor under software control?
    The governor is the last-gasp step in between the software-controlled parts (a sort of secondary throttle restricter, except that it has veto power over excessively enthusiastic use of the throttle) and the actual engine. Its function is to protect the engine from the software. Really, it isn't hard. You just need to let the hardware designers express the paranoia they should automatically feel when someone says, "the software can take care of that."

    EDIT: If the solenoids are on the fuel injectors for some reason, you give the governor authority over, say, the incoming fuel supply, or, say, a butterfly valve in the intake manifold after the injectors.

  • (disco)

    Overall, you design the hardware so that it can protect itself against the software.

  • (disco)

    ...and the true origin of Stuxnet is revealed

  • (disco) in reply to Steve_The_Cynic
    Steve_The_Cynic:
    If the solenoids are on the fuel injectors for some reason

    Call me paranoid, but this is one of the reasons I keep some plastic bags in my truck. If it ever experiences an engine runaway, I can wrap the bag around the air intake and hopefully starve the engine before it melts itself down.

  • (disco) in reply to Steve_The_Cynic
    Steve_The_Cynic:
    Seriously, never, ever rely on software to protect hardware. Software has bugs, and if the hardware doesn't have protective elements in it, the bugs can damage the hardware, or even kill people.

    Never rely on software to protect people. It's sometimes less effective (cost-wise or in overall downtime) to protect hardware from all possible breakage-inducing faults with more hardware.

    Software has bugs, true, but hardware wears out and is greatly affected by the quality of the maintenance team and their practices.

  • (disco)

    TR :wtf: is not checking for integer overflow in such critical software:

     const uint16_t max_pos = min_pos + m_movement_range;
 vaeAssertRelease(max_pos >= min_pos);
  • (disco) in reply to manofsteel76667
    manofsteel76667:
    Steve_The_Cynic:
    Seriously, never, ever rely on software to protect hardware. Software has bugs, and if the hardware doesn't have protective elements in it, the bugs can damage the hardware, or even kill people.

    Never rely on software to protect people. It's sometimes less effective (cost-wise or in overall downtime) to protect hardware from all possible breakage-inducing faults with more hardware.

    Software has bugs, true, but hardware wears out and is greatly affected by the quality of the maintenance team and their practices.

    Up to a point, I agree. On the other hand, engine overspeed protection (the example cited above) protects people, by working to protect us against broken bits of engine that are ejected from the engine compartment, so it's probably worth doing.

    And the simplest of all hardware-to-protect-hardware is so cheap that it is ubiquitous: the electrical fuse.

  • (disco)

    The real WTF is the syntax highlighting fail:

    http://i.imgur.com/QGMD1mi.png

  • (disco) in reply to Protoman

    The correct way is obvious:

    void PlcPart::set_target_position (uint16_t parPosition) {
 using boost::algorithm::clamp;
 const uint16_t min_pos = m_set_base_position;
 const uint16_t max_pos = min_pos + m_movement_range;	
 m_target_position = clamp(parPosition, min_pos, max_pos);
}
  • (disco) in reply to Steve_The_Cynic
    Steve_The_Cynic:
    EDIT: If the solenoids are on the fuel injectors for some reason, you give the governor authority over, say, the incoming fuel supply, or, say, a butterfly valve in the intake manifold after the injectors

    The "for some reason" is because you need to adjust the amount of time the fuel injectors are open for so you can adjust the mixture for different speed and load conditions. They're mounted right above the inlet valves so there's as little distance as possible for the fuel to travel between being injected into the inlet tract and being sucked into the cylinder. If you make that path longer, the engine will stumble and splutter when you open the throttle quickly and then flood with a huge surge of unburnt fuel out the exhaust when it finally catches up. So, introducing a second butterfly between the injector and head would increase the intake path and make the engine run badly.

    In practice what happens is that if the engine overspeeds, the ECU cuts off the fuel entirely by just not opening the injector solenoids. This never, ever goes wrong, because it's part of the map that describes how much fuel should be added at different engine speeds - if it went wrong, the engine wouldn't run at all. Adding a mechanical governor and a whole bunch of other stuff would make it less reliable, heavier and more expensive to produce, and most likely not actually work as well as setting a limit in the ECU software.

  • (disco)

    Never ever trust software to protect your equipment. Use limit switches and other sensors hard wired into the control circuitry.

    Even if the code was for some embedded controller rather than a PC the same criteria apply.

    That said, numerical limits (if A > B then A = B) are always a good idea, optionally raising a flag.

    Trusting a PC to directly interface with the machinery is a really bad idea, it will crap out at the most inconvenient moment. Use a PLC, dedicated hardware with a specialized programming language, and interface that with the supervisory on the PC. If communication is lost the PLC can detect it and take the necessary measures.

  • (disco) in reply to gordonjcp
    gordonjcp:
    Steve_The_Cynic:
    EDIT: If the solenoids are on the fuel injectors for some reason, you give the governor authority over, say, the incoming fuel supply, or, say, a butterfly valve in the intake manifold after the injectors

    The "for some reason" is because you need to adjust the amount of time the fuel injectors are open for so you can adjust the mixture for different speed and load conditions. They're mounted right above the inlet valves so there's as little distance as possible for the fuel to travel between being injected into the inlet tract and being sucked into the cylinder. If you make that path longer, the engine will stumble and splutter when you open the throttle quickly and then flood with a huge surge of unburnt fuel out the exhaust when it finally catches up. So, introducing a second butterfly between the injector and head would increase the intake path and make the engine run badly.

    In practice what happens is that if the engine overspeeds, the ECU cuts off the fuel entirely by just not opening the injector solenoids. This never, ever goes wrong, because it's part of the map that describes how much fuel should be added at different engine speeds - if it went wrong, the engine wouldn't run at all. Adding a mechanical governor and a whole bunch of other stuff would make it less reliable, heavier and more expensive to produce, and most likely not actually work as well as setting a limit in the ECU software.

    Right, so here you are saying that basic operation of the machine imposes a very strict software reliability requirement. The software load is also relatively small (compared to, say, an application like Excel), and more amenable to strict development quality controls, NASA-style.
  • (disco)

    Two words: Fail Safe

  • (disco) in reply to perkelemonkey
    perkelemonkey:
    Even if the code was for some embedded controller rather than a PC the same criteria apply.

    The computer system that is directly responsible for hardware control needs to be subject to a safety critical quality testing regime; it is responsible for ensuring that the hardware system runs safely whatever inputs it is given. The computer system in question is probably not going to be a conventional desktop computer (except when dealing with hobbyist systems).

    Once you've got safe operation enforced, the rest of the code doesn't need to be nearly as highly tested.

  • (disco)

    TRWTF is no comment to describe the details.

  • (disco) in reply to Steve_The_Cynic

    Well, to an extent. There's a lot going on, but it's a thing that's fairly easy to test. In this case any mechanical device that you try to apply will have pretty hideous effects both in normal running and in failure conditions, and having the thing slope off sharply to "no fuel" above a certain speed is a simple emergent property of the design of the ECU.

  • (disco) in reply to mott555

    Diesel engine? Sticking a plastic bag over the intake won't work, it'll just suck the bag in along with your hand. You need something fairly rigid that won't bend. The simplest method is just to put your foot on the brake, put it in gear, and gently let the clutch up until it stalls.

    If it's run away, there's usually a good reason, quite often fuel getting into the oil until it starts to get sucked up through the crankcase breather or oil getting sucked in through failed turbocharger bearings. Both are caused by a lack of maintenance. Either way, the engine's been run at high speed while starved of oil and is probably never going to work right again.

  • (disco) in reply to gordonjcp
    gordonjcp:
    Diesel engine? Sticking a plastic bag over the intake won't work, it'll just suck the bag in along with your hand. You need something fairly rigid that won't bend. The simplest method is just to put your foot on the brake, put it in gear, and gently let the clutch up until it stalls.

    My air filter will not pass a plastic bag. This isn't exactly it, but it's something like this, and a plastic bag covers it pretty well. Also, it's an automatic, so I can't stall it out.

    [image]
    gordonjcp:
    oil getting sucked in through failed turbocharger bearings.

    This is what normally causes a diesel runaway, but I have an old LB7 Duramax and those things are notorious for injector failure. They often fail and stick open, though I guess the chances of enough of them failing at once to runaway the engine are pretty slim.

  • (disco) in reply to mott555

    Never liked K&Ns, they're really popular on Rover V8s but they never seem to run right with them fitted.

    Isn't the Duramax conventional injection rather than common-rail? If the injector starts leaking you'll get holey pistons rather than runaway.

  • (disco) in reply to dkf

    The code may be tested 100% safe, but you still have to worry about the controller hardware. An input or output may short out and be always ON, with possibly dire consequences.

    Which is why safety critical sensors must be hardwired into the system, turning off command power and other supplies.

    Skimping on safety may easily result in million dollar machines stopped waiting for the bent metal to be replaced. And worse.

  • (disco) in reply to loose

    Yeah, I've read that book.

  • (disco) in reply to gordonjcp

    Mine's not actually a K&N, I don't recall what it is but it's similar.

    It's common-rail injection, I think all diesel pickups made in the past 30 years are. Holey pistons have happened too, though not to me.

  • (disco) in reply to Steve_The_Cynic
    Steve_The_Cynic:
    by working to protect us against broken bits of engine that are ejected from the engine compartment, so it's probably worth doing.
    I remember now what it was I was thinking of when I wrote this, besides charming phrases like "uncontained turbine failure". It was a car owned by someone my wife knew when she was in the (US) Air Force(1) - this person was immensely distressed when his car ejected the fan from the engine compartment *through* the (closed) hood.

    (1) Members of the audience who've been following my posts will have noticed that I'm English and that I live in France. Those with longer memories may even remember that I lived for some years in the US. I met her there, after she had left the force.

  • (disco) in reply to Steve_The_Cynic

    Way back in ancient times I was called out to fix a large numerically controlled drill-press, the kind that had a turnstile head with like 30 different drills. It had this platform that huge castings could be positioned on, to the thousandth of an inch. The platform was moved in X and Y by this large 30 horsepower hydraulic actuator. They called me in because the platform was going "loopy", at certain x,y locations it would oscillate a few tenths f an inch, at other places it would be steady, and at a few others it would oscillate like THREE FEET, at about 20 times a second. The sight of a many-ton platform vibrating at that rate was awesome. Even more awesome, the shop owner didn't beleive in safety guards, so he was missing a couple of fingers already and the platform also had no safety barricade around it, so if you were standing right by the platform and it decided to start vibrating, you might end up in two pieces.

    I declined this fabulous opportunity to fix the thing.

  • (disco) in reply to george_gonzalez

    Yikes! (And I'm glad to hear that you passed that one)

    I am a safety officer, and hearing stories like that one give me the shivers.....

    I also program control systems for mass flow controllers (for gasses like hydrogen and carbon monoxide etc), furnace controllers etc, but I do under no circumstances allow our systems to rely on my programming skills for human safety (as I can screw up as well as the next guy). That has to be handled by autonomous systems like (safety) PLC's or normal hardware interlocks etc....

  • (disco) in reply to Steve_The_Cynic
    Steve_The_Cynic:
    this person was immensely distressed when his car ejected the fan from the engine compartment through the (closed) hood.
    Why? He wouldn’t have much to fear from it, given that the engine fan in a car tends to be mounted vertically :)
  • (disco) in reply to Gurth

    Well was it me, I would have been annoyed that I didn't get it on video :smile:

  • (disco) in reply to sehetw

    Clbuttic!

  • (disco) in reply to RFoxmich

    Mine's in imperial units, so it goes to 12!

    Onyx you've led a culturally deprived life. Here is the scene from "This is Spinal Tap" I was paying homage to:

    https://www.youtube.com/watch?v=KOO5S4vxi0o

  • (disco) in reply to mott555
    mott555:
    If it ever experiences an engine runaway, I can wrap the bag around the air intake and hopefully starve the engine before it melts itself down.

    If you have access to the engine compartment, you could also do things like rip out relays or disconnect the battery.

  • (disco) in reply to Steve_The_Cynic
    Steve_The_Cynic:
    Right, so here you are saying that basic operation of the machine imposes a very strict software reliability requirement. The software load is also relatively small (compared to, say, an application like Excel), and more amenable to strict development quality controls, NASA-style.

    Right up until someone buys a 3rd-party customized ECU, right?

  • (disco) in reply to FrostCat

    Diesels are largely mechanical. If one experiences a runaway I don't think there's anything you can do to the electronics to stop it. There's no ignition system to disconnect, and usually they're running on oil leaking from the turbo so stopping the fuel injectors or fuel supply will have no effect.

  • (disco) in reply to mott555
    mott555:
    . If one experiences a runaway I don't think there's anything you can do to the electronics to stop it

    Ah. My experience with diesels is very limited, as in "I drove a diesel Olds 88 in the 80s a few times".

  • (disco) in reply to mott555
    mott555:
    usually they're running on oil leaking from the turbo so stopping the fuel injectors or fuel supply will have no effect.

    I don't know if I should be horrified that an engine can be so easily switched over to alternative fuels or bedazzled from how awesome diesels are that they are so easily capable of it!

  • (disco) in reply to Tsaukpaetra
    Tsaukpaetra:
    I don't know if I should be horrified that an engine can be so easily switched over to alternative fuels or bedazzled from how awesome diesels are that they are so easily capable of it!

    Diesels can run on anything that's oily and not solvent-y (like gasoline). I've considered doing the used fast food cooking oil thing before.

  • (disco) in reply to perkelemonkey
    perkelemonkey:
    The code may be tested 100% safe, but you still have to worry about the controller hardware. An input or output may short out and be always ON, with possibly dire consequences.

    Sensor failure is usually fairly easy to detect if you're looking for varying patterns rather steady states. Level-sensitive stuff needs to be checked (and recalibrated) more often. For the actuator side, you also really want systems that will fail to doing nothing when the signal stops changing. Like that, if systems fail into any steady state, you either detect it or at least stop making things worse.

    Some systems (e.g., train brakes) may need additional safety measures. You'd want those with mechanical control too. However, my main concern is how in some cars all the braking systems are under the control of the ECU. If shit goes wrong, how can we be sure that we can stop safely? I'd much rather have the handbrake as a separate manual-only system; sure the ECU is allowed to detect the state, but it shouldn't be able either engage or disengage it. Yet that's no longer the world we live in: better trust that some programmer got it right for once…

  • (disco) in reply to gordonjcp
    gordonjcp:
    So how do you get that to reliably control the speed of the engine, where the fuelling is done by a bunch of solenoids under software control and the throttle position is set by a stepper motor under software control?

    Valve bounce will do it. Use the appropriate springs on the valves and you have an effective engine speed limiter in hardware.

  • (disco) in reply to mott555
    mott555:
    Call me paranoid

    You're paranoid. Runaways aren't due to software failure, they're usually because of a mechanical failure or leak supplying fuel or oil. I think with modern vehicles they're also exceedingly rare, but I could be wrong on that.

    mott555:
    If it ever experiences an engine runaway, I can wrap the bag around the air intake and hopefully starve the engine before it melts itself down.

    I've had the same thought, but my intake isn't so easy to get to, and impossible to get around. It's likely to just ingest anything I try it with, which I suppose will at least stop the runaway but will leave a bit of a mess to clean up. Easier to just stall it. Yet another reason why manual transmission is superior to automatic.

  • (disco) in reply to Steve_The_Cynic
    Steve_The_Cynic:
    charming phrases like "uncontained turbine failure"

    This brings to mind lovely images of tiny high-speed circular saw blades flying through the air. Sounds like something straight out of a B-grade horror movie.

  • (disco) in reply to mott555
    mott555:
    Diesels can run on anything that's oily and not solvent-y (like gasoline). I've considered doing the used fast food cooking oil thing before.

    Rudolf Diesel had a patent that described an engine that ran on coal dust.

Leave a comment on “A Hardware Switch”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article