• someone (unregistered)

    "one man-day per day" = one man.

  • Little Bobby Tables (unregistered)

    Just another day at the office.

  • RLB (unregistered) in reply to someone

    Not in this case, it wasn't.

  • gknu (unregistered)

    "until the systems they monitored were finally retired"

    Hah. good one

  • dpm (unregistered) in reply to someone

    More like "Two man-days per man". One man-day is still considered only eight hours, and they were coding around the clock, crashing on air mattresses when necessary.

  • (nodebb) in reply to dpm

    The "man-day per day" was referring to the two men working from 6AM-10AM (4 hours apiece x 2 = 8 hours/day).

    "Someone" pointed out that this is just the same as "one man". They had the equivalent of one FTE doing nothing but verifying batches.

    Glad all us programmers could clear that up.

  • Raj (unregistered)

    Dear diary,

    Today at work I wrote poorly designed shell scripts with hard-coded things in them to monitor poorly written batch jobs. They were so cool they generated a bit-mask. We gave a copy of those scripts to other people and it caused problems. What an unpredictable outcome!

  • Jim B (unregistered)

    To those who have not worked in a large financial institution stories like this seem nonsensical. I've been there - they are real WTF's.

    I worked for 21 years for a brokerage firm that was acquired by a bank in the mid 2000's. The bank imploded a years later and was acquired by another bank during the meltdown of 2008. So the brokerage firm I worked for was now part of a very large bank. Ugh.

    The brokerage firm had a brilliant idea in the 2001-2003 time frame to replace their mainframe trading system with a Unix based system (on Tandem's - remember them). This was a total failure. So the broker went to a third-party processor (which lead indirectly to being bought out as bank #1 used the same back office system).

    For some management 'reason' bond trading was not bought as part of the 3rd party trading system. So, bypassing IT, the bond department contracted out to develop their their own trading system. All was supposed to be OK with this system through testing. Problem was there had never been a full sized end-to-end test of the system which interfaced with the main system. All tests were with 10 or so branches sending a limited amount or orders. The problem was when all 950+ branches were brought online that the system dragged to a halt. No one at the high-priced contracting firm had ever thought of that.

    So, in order to keep the business running people from all over the firm were brought in to manually enter bond orders. The old system was brought back up which took a week. The new trading system was re-written in-house after this and (gasp) fully tested before moving to production.

    Was anybody fired over this? Nah, bond management (who made the original decision) made out very well when the firm was 1st bought.

  • (nodebb)

    "everyone has root". I was expecting to read where this hit the fan.

  • Dinosaur (unregistered)

    If they had stayed on the mainframe, there are at least two products to do exactly this production plan/check/repair. But that wouldn't have been as much fun for the managers.

  • (nodebb) in reply to Raj

    Today at work I wrote poorly designed shell scripts with hard-coded things in them to monitor poorly written batch jobs. They were so cool they generated a bit-mask. We gave a copy of those scripts to other people and it caused problems. What an unpredictable outcome!

    This ^^ is TRWTF ^^. The script should have been sanitized, and all the "cause harm to my own system" parts should have been sanitized AND commented out so nothing could run.

    Hey, just thought of another WTF. All 9 "trading systems" ran on the same servers? How did any of these systems have access to the other systems' data and other files??

  • Super Cup Final (unregistered)

    https://livewatchgay.de/realmadridvsatleticomadrid/ https://livewatchgay.de/realmadridvsatleticomadrid/

  • siciac (unregistered) in reply to Bananafish

    The script should have been sanitized, and all the "cause harm to my own system" parts should have been sanitized AND commented out so nothing could run.

    Right. Think about how this script came about:

    The first thing they had me do was learn how to verify that all 87 (yes, eighty seven) of the nightly batch jobs had completed correctly. For this task, both the team manager and lead dev worked non-stop from 6AM to 10AM - every single day - to verify the results of the nightly jobs. I made a list of all of the jobs to check, and what to verify for each job. It took me from 6AM to 3:00PM, which was kind of pointless as the markets close at 4PM.

    Not validating and sanitizing is what happens when an engineer runs out of fucks to give.

    Hey, just thought of another WTF. All 9 "trading systems" ran on the same servers?

    That was addressed in the story:

    All bureaucracy was abandoned in favor of: everyone has root in order to do whatever it takes on-the-fly, no approvals required.

  • Gerry (unregistered) in reply to Jim B

    Everything is fast for small values of n

  • Darkenon (unregistered) in reply to Gerry

    Not true. I've seen algorithms that infinitely loop when n is 0.

  • fintech guy (unregistered)

    Somehow this sounds strangely familiar. lots of nightly batch jobs and cryptic error messages requiring manual fixes is all in a days work at my small bank. And the worst thing is the system is too big, complicated and important to replace with something modern :(

  • Bob (unregistered)

    +1 Raj

  • root (unregistered)

    Your problem was using ksh instead of bash.

Leave a comment on “A Shell Game”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article