• Pjrz (unregistered)

    Ah, the old "this is the way it has always been and it CANNOT BE CHANGED" kind of manager. Together with a dose of misdirected sarcasm. Not a good combination.

  • bvs23bkv33 (unregistered)

    then another instance of ITAPPMONROBOT was spawned

  • Pyth0n (unregistered)

    OMG, I had console like one on the photo... I'm sooooo ooooold...

  • (nodebb)

    Did he leave him any ping pong balls to go with the paddle? Oh no, he probably tried ordering using the requisition system but the request kept going to a warehouse which doesn't have them in stock.

  • Pumuckel (unregistered)

    4.512px × 3.000px and 5.5MB, who knows, perhaps someone with a 16k monitor is reading these articles. How about not including Wikipedia images directly, but making a smaller thumbnail instead?

  • (nodebb)

    Was Jacob another non-programming programmer? You know the type. Everything's too hard or would take too much time, even if it's already done.

    I would be tempted to make him eat that paddle.

  • Kleyguerth (github)

    I'm just wondering: when did "server crashing bug" become "technical debt"?

  • Brian (unregistered)
    We have too much else to do. We can't be worrying about technical debt like this.

    Alright, that made my head twitch a little bit, because it's both scary and all-too-familiar.

  • (nodebb) in reply to Kleyguerth

    Around the same time that people who weren't shot became victims and college students started getting PTSD. On a serious note, this white washing of technical problems is really annoying. My pet peeve is the term "micro optimization". Today we have operating systems that need 16 cores and 32GB RAM just to boot up, because along the way every inefficiency was deemed a micro optimization not worth the effort to address.

  • (nodebb)

    Easy fix: compute what the 95-percentile of how many items of that particular piece of inventory are dispatched by each warehouse each day (the historical data ought to be available in the DB backups!) and only dispatch from those warehouses that have more than that remaining. Of course, that does mean that all the warehouses are going to need to hold a lot of inventory in order to get traffic, but that's a price well worth paying.

  • Herr Otto Flick (unregistered)

    Sometimes shit just wears you down. I have a colleague who still works in the same company, he's been here a long time (less then Herr Flick, obvs). When he started, he was idealistic, tried to fix things, was proactive. Problem was, he was typecast as "the .NET guy", and we just have one .NET project, so he worked the same thing forever and ever, until it just wore him down. Now, he will spend all day arguing why we cannot do anything, rather than trying to do something.

    Herr Flick knows the game, change team every 2 years maximum, keep learning new skills, keep fresh. Evolve, adapt, overcome.

  • King (unregistered)

    So: System crash, system rebooted, orders lost, problem solved?

  • Brian Boorman (google)

    All seems reasonable to me. You can look at the world with idealistic-everything-is-black-and-white glasses on, but news flash - it's not. Sometimes you have to make priority calls about what is more important. If this doesn't happen all too often, is it more important than something else they need to roll out? Maybe not. Perhaps the contract that they were hired under doesn't allow for fixing things that are specifically called out in the contract.

    Also, don't forget - this is government workers getting supplies from a government warehouse. It's not like a company is losing orders when the user gets it elsewhere - the items are already bought and paid for and sitting there. The requisition system just helps get it to you and report whose budget line gets the charge when accounting balances the books.

  • Geoff (unregistered)

    I don't understand someone designed system that transmits orders realtime but syncs inventory only once a day? I don't see how the ping-pong could happen more than once a day and crash the system otherwise...

    If the orders are being sent realtime why not send an updated inventory-item-count message or something iff the order is accepted? Send the delta and an expected result, so you can raise an exception if the counts differ? I mean its like nobody was even thinking..

  • robby the robot (unregistered)

    System crashes, someone restarts it, everything is working again. So possibly it's a few minutes a week. Or be all idealistic and spend two months rewriting everything to prevent that crash, and another couple of months fixing the new bugs, and then add on all the time that your changes have delayed any other changes which were due to be added.

  • (nodebb)

    New and trivial crap "wouldn't it be nice to have" features dreamed up by some business type are important top priority tasks because the requestor signs the checks/approvals for projects.

    Keeping the system from crashing due to fundamental flaws? Irrelevant "technical debt" not important enough to do anything but waste time recovering from them.

    Yup - business as usual at most orgs (not just the government).

  • (nodebb) in reply to Developer_Dude


    Way too many developers overestimate the difficulty of changes because they aren't very smart and proceed to broadcast that fact by wasting more effort on making excuses than it would've taken to solve the problem.

  • Government Service Survivor (unregistered)

    That's how Mark learned the most important lesson of working in the government:

    "Never be first. Never be last. Never volunteer for anything."

  • badwiring (unregistered)

    If you order something, one database says it's available, but the other says it's not, the system crashes. After the system has crashed and restarted, doesn't the person who originally needed to order something still need to order it? Do they know that their order crashed the system? Wouldn't they just try placing the order again after the system was restarted? Or are they supposed to just know somehow not to do that and wait until the next day? What if they really need whatever it is? Doesn't that mean the opposite is also true - people are waiting for an item, you have it, but nobody knows you have it because the databases haven't been synced? Or you order a product that's available in a nearby warehouse and it ships from farther away, so you wait longer to get it than if you had waited a day.

  • Decius (unregistered)

    "The order system crashed, get Mark to restart it." "Mark is offsite, we need to get someone else to restart it." "The order system crashed, get Mark to restart it."

  • Chuck (unregistered)

    I may have worked on that system at one point....

    What Mark isn't taking into account is that the system has been in use since the 70's. the term, 'Spaghetti Code' doesn't really describe it, it's more a writhing mass of tentacles that has evolved over the course of 50 years of lowest cost contract cycles.

    When Manager Jacob talks about the technical debt, what he really means is that he is weighing Mark's simple change against the chance that an unintended side effect will cause one of the depots to ship all of their titanium widgets to BFE at a cost of ONE BRAZILLION DOLLARS.... which would draw potentially career ending attention to his division from the powers that be.

  • Guest (unregistered)

    Obviously the right thing to do is prevention. Create automation to ensure warehouses have enough in stock based on the demand of the previous day. Then you prevent the denial of an order, right?

  • (nodebb) in reply to Decius


  • (nodebb)

    The correct answer from Mark: "I'd love to do it but it is not in my project description. I'd be happy to renegotiate the contract, however"

    If he can't say that, than he's TRWTF.

  • medievalist (unregistered)

    Personally I would have just fixed it, without permission, and moved on. I flunked the Milgram experiment.

    Of course, that's is why some managers love me and some managers hate me, but nobody's indifferent!

  • robby the robot (unregistered) in reply to medievalist

    I'm currently working with someone like that - a problem gets reported, he jumps in and fixes it. But doesn't tell anyone that he's doing it, look at the work anyone else is doing in the area, or pick up the pieces when it breaks other things.

  • Barf4Eva (unregistered) in reply to robby the robot

    hahaha, yeah for sure this!

    Another way to rewrite medievalist's post that takes your post in to account...

    "Of course, that's is why some developers love me and some developers hate me, but nobody's indifferent!"

    And there is truth to that. Some are gonna love ya for it. Some are gonna hate ya for it. Another phrase that I've heard a lot is.. "Don't ask for permission, beg forgiveness later" lol

  • (nodebb)

    Communication and respect take you a long way in this world.

  • joderor (unregistered)

    what's even stupider about this developer is that he's a contract resource...so (ideally) he was hired to perform a specific scope of work...and instead starts finding other stuff to do. ridiculous. no wonder the manager was exasperated.

    "but, but, but...the server crashes"...

    WHO GIVES A FUCK!?! that in and of itself doesn't matter...the business (government) impact is what matters here and they've deemed the existing situation to be sufficient.

  • eric bloedow (unregistered)

    reminds me of several stories where two people accidentally set up their e-mail forwarding function in a way that bounced one e-mail back and forth until an e-mail sever ran out of space, blocking the WHOLE COMPANY from using e-mail...

  • DavidTC (unregistered)

    Gah, this is dumb. Let's work with the idea that we can't actually save any state with the order, but there are still obvious solutions:

    1. Stop resubmitting the requests so fast. Yes, it's very stupid to keep doing them over and over at all, but doing it once every five minutes would turn this from 'completely idiotic' to just 'mostly idiotic'.

    2. The routing engine should roll the dice every request and 1 time out of 1000, it routes the order to the second or third closest warehouse instead of the closest.

    3. Even if the system has to be synced each night, okay, but what exactly is stopping the ordering end from also keeping track? Each day it starts with the right amounts, and it reduces it by one each order., before getting reset the next day.

    The brilliance of this is that it works even if there are multiple ordering locations, because it would, at least, stop the idiotic infinite loops, assuming it decremented 1 each time it tried to order. It might have to loop through a few dozen of times, but it would eventually hit zero on that warehouse.

Leave a comment on “A Mean Game of Pong”

Log In or post as a guest

Replying to comment #:

« Return to Article