• Nicholas (unregistered) in reply to Biff

    I typically work on multi-hundred-KLOC systems, including compilers and molecular simulators. My code usually has few problems. The problems I run into are usually subtle race conditions or failure to observe zero-copy semantics.

  • (cs) in reply to Dlareg
    Dlareg:
    I'm so happy I'm the only user of my programs :D So I'm only annoying my self when shit happens.

    But the real question is, do you yell at yourself, "DEPLOY, DEPLOY, DEPLOY!!!"

  • Mike L (unregistered)

    Wow, and I thought that my Tuesday morning, crashing 120 stores worth of registers, was a bad day.

    I guess the conflict between operations (who want is now) and development (who want it later) is universal.

    But... I'm sure the developer had lot of help, with operations people giving sound advise like "just fix it".

  • Ben (unregistered)

    Not much of a WTF. When losing tens of thousands (or more) per minute, the more cost-effective solution may very well be to bypass rigorous QA.

    The real WTF is that this sounds like a problem that a proper automated test suite would have caught. Not to mention, as another poster noted, there was apparently no DR plan or ability to roll back the recent update. That is just inexcusable in such an environment.

  • Vlad (unregistered)

    A very good example of detrimental impact of management on software development. It usually happens in the long run, since no manager in his mind would care about what will happen to the software a year from now; but this case is really hilarious: seconds count, minutes don't.

  • S (unregistered) in reply to Ben

    They lost the tens of thousands because they bypassed QA. This introduced the bug

  • Vlad (unregistered) in reply to Ben

    But they have no time to create and DEPLOY those automated tests. :)

  • DFK (unregistered) in reply to Nicholas
    Nicholas:
    You know, the real problem in this scenario is with the lousy developers. They should just frickin' get it right the first time. I don't know how many hundreds of people I've met who really just can't be trusted to code. Use your brains, people! Don't call it done if it isn't done! Think about *all* the code paths! Learn the system you're working with. And don't whine about not having the time. Just remember this mantra, which has always served me well: no matter how much you think it's somebody else's fault, it's really your fault. And your problem. You have no idea the world of problems this attitude will fix, and how far it will take you. Though it may just make you frustrated with everybody else's incompetence.

    Or worse. They decide to gang up on you for showing them up . Nothing is worse than a worse-than-failure situation in an environment of managers stacked one on each other all afraid of failure. You can set off time bombs that way.

  • Jeff (unregistered)

    I tried to get my dog to do logarithms, only to realize I don't have a dog.

    Thus, he's doing quantum mechanics.

    My cat is composing poetry...by carving it into the couch. It's great poetry, but sadly written in Cat. It'll take some time to translate.

    CAPTCHA: nobis

    I think it means 'no biscuit' for my dog.

  • mjmcinto (unregistered) in reply to QuinnFazigu
    QuinnFazigu:
    Archibald Buttocks:
    And this is why I stopped developing trading software - it's a fast-track to an early stress-induced death, and the pay is nothing short of dire.

    Is the pay really that bad? I'd imagine they'd pay top dollar for bearing such horrible stress. A mistake at our business (real-time service, but each transaction is small) can cause money to bleed, but nothing like the stock market.

    I think a lot of it depends on where you work. I worked as a developer for a trading company, and sat on the floor w/the traders. The pay was on the high side of average, but the bonuses were definitely above average (as long as the company did well of course)

  • CC Pizzabutt (unregistered) in reply to Archibald Buttocks

    When I started out of school (2004), the starting pay was $65K for 0 years of experience. This was at one of the bigger banks, perhaps the smaller firms don't pay nearly as well. ~3 years later, and I'm making double that almost.

  • Chucara (unregistered)

    This must be the real time programming our professor in Java class was always talking about.. Nice to see it in action!

  • Paula (unregistered)

    1.) Deploy. 2.) ????? 3.) Profit, Bitches!

  • Anon (unregistered) in reply to Archibald Buttocks
    Archibald Buttocks:
    WallStreetDeveloper:
    I was surprised he said the pay was dire. Where else can you earn $200k+ as a developer?

    Not developing trading software! Maybe it's different in the UK, but the highest paid guys in the company were getting $80k, tops.

    Working in NYC, a good developer at a financial firm should easily be in the $150k+ range.

  • Lyle (unregistered)

    I deployed first. In fact, I deployed before the software was even written.

  • John Thomas (unregistered)

    the "market" scares the heck out of me, especially during tough economic times where anything can happen.

    JJ http://www.Ultimate-Anonymity.com

  • KonAitor (unregistered) in reply to Ben
    Ben:
    Not much of a WTF. When losing tens of thousands (or more) per minute, the more cost-effective solution may very well be to bypass rigorous QA.

    The real WTF is that this sounds like a problem that a proper automated test suite would have caught. Not to mention, as another poster noted, there was apparently no DR plan or ability to roll back the recent update. That is just inexcusable in such an environment.

    You people should try reading the whole post. The OP specifically said that because the market had opened it was to late to rollback...

  • jafraldo (unregistered) in reply to NaN

    you mean to say he can fetch logorithms! Or catch them!

  • A. Developer (unregistered)

    If you get into a situation where the boss is demanding untested functionality in production, then push back. Write a brief but direct e-mail, like follows:

    Mr. Boss:

    I understand the timeline for feature X is crashed and that feature X is on the critical path for our current project.

    Feature X has not gone through our standard procedures for review, testing, or deployment. Defects may exist in feature X that have not been discovered and therefore cannot be removed. Without standard procedures, these undiscovered defects may evidence themselves in production. This may impact our systems, possibly in catastrophic ways.

    Are you absolutely sure you want to deploy feature X to production without putting it through the standard procedures?

    A. Developer

    If you get back a "Yes", then kick your feet up. The responsibility lies entirely with the boss.

  • dave (unregistered) in reply to rumpelstiltskin
    rumpelstiltskin:
    NaN:
    taylonr:
    Any objections raised by the developers were met with the same reaction you’d get from a dog after explaining the Pythagorean Theorem – blank, drooling faces.

    You have a pretty stupid dog

    Yeah, my dog does logarithms now.

    I'm having problems getting it to learn English though, so he can't tell me his answers.

    You can't teach a dog to "tell" you the answers, because his speech system is too primitive. You have to teach him to write the answers down.

    Problem is this only works in the winter. With a boy dog.

  • Joe (unregistered) in reply to Nicholas
    Nicholas:
    You know, the real problem in this scenario is with the lousy developers. They should just frickin' get it right the first time. I don't know how many hundreds of people I've met who really just can't be trusted to code. Use your brains, people! Don't call it done if it isn't done! Think about *all* the code paths! Learn the system you're working with. And don't whine about not having the time. Just remember this mantra, which has always served me well: no matter how much you think it's somebody else's fault, it's really your fault. And your problem. You have no idea the world of problems this attitude will fix, and how far it will take you. Though it may just make you frustrated with everybody else's incompetence.

    I'm guessing the most complex application this guy's ever worked on was "Hello World!".

  • Franz Kafka (unregistered) in reply to Joe
    Joe:
    Nicholas:
    You know, the real problem in this scenario is with the lousy developers. They should just frickin' get it right the first time. I don't know how many hundreds of people I've met who really just can't be trusted to code. Use your brains, people! Don't call it done if it isn't done! Think about *all* the code paths! Learn the system you're working with. And don't whine about not having the time. Just remember this mantra, which has always served me well: no matter how much you think it's somebody else's fault, it's really your fault. And your problem. You have no idea the world of problems this attitude will fix, and how far it will take you. Though it may just make you frustrated with everybody else's incompetence.

    I'm guessing the most complex application this guy's ever worked on was "Hello World!".

    I'd say it was a troll, but it isn't even that. Nick is just being sarcastic.

  • (cs) in reply to KonAitor
    KonAitor:
    Ben:
    Not much of a WTF. When losing tens of thousands (or more) per minute, the more cost-effective solution may very well be to bypass rigorous QA.

    The real WTF is that this sounds like a problem that a proper automated test suite would have caught. Not to mention, as another poster noted, there was apparently no DR plan or ability to roll back the recent update. That is just inexcusable in such an environment.

    You people should try reading the whole post. The OP specifically said that because the market had opened it was to late to rollback...

    Erk.

    Erk erk erk.

    Please do not ever work anywhere near an OLTP system.

  • Tom Steele (unregistered) in reply to powerlord

    [/quote] It could be worse... it could be Schrödinger's cat![/quote]

    Then it would be right as long as you didn't look at it.

  • (cs) in reply to powerlord
    powerlord:
    It could be worse... it could be Schrödinger's cat!
    Schrödinger's cat is actually in her box under my desk doing vector calculations.. or maybe she isn't. Gee, I better open this box to see what she's up to...
  • (cs) in reply to A. Developer
    A. Developer:
    If you get into a situation where the boss is demanding untested functionality in production, then push back. Write a brief but direct e-mail, like follows:

    Mr. Boss:

    I understand the timeline for feature X is crashed and that feature X is on the critical path for our current project.

    Feature X has not gone through our standard procedures for review, testing, or deployment. Defects may exist in feature X that have not been discovered and therefore cannot be removed. Without standard procedures, these undiscovered defects may evidence themselves in production. This may impact our systems, possibly in catastrophic ways.

    Are you absolutely sure you want to deploy feature X to production without putting it through the standard procedures?

    A. Developer

    If you get back a "Yes", then kick your feet up. The responsibility lies entirely with the boss.

    This is the correct way to handle this situation. However, management might interpret this as the developer's way to deflect responsibility for doing a sloppy job. It gets worse when you have to write this email more than once or twice per year. If you're always being put on the spot to deploy untested software and take the heat for its failure, then keep writing those emails, but look for a new job.

  • omx broker employee (unregistered) in reply to Kaos

    To be fair, the Saxess software had actually been tested. There was a market wide acceptance test about a month ago that went off without a hitch, and when the software was deployed on Saturday, all members were required to log into the system as they would on a normal day. That, too, worked well enough that Nasdaq OMX announced the final "go" decision.

    Who knows why it refused logons today and yesterday. Maybe it will be revealed in the fullness of time, but it wasn't that the system was deployed untested. Unless of course they deployed a different system after everyone had reported success on the first deployment on Saturday.

    Why they didn't roll back today, I don't know. There were procedures in place, I believe (I wasn't directly involved).

  • Won't Get Fooled Again (unregistered)

    "After running some numbers, Daniil discovered that the amount of lost commissions, sales, and trades outweighed the cost of testing and fixing the change over ten-fold. ... "

    Yeah, right.

    All these fake Daily WTF stories always have a "tell" the makes it obvious it was either made up wholesale or embellished beyond all recognition.

  • (cs) in reply to Won't Get Fooled Again
    Won't Get Fooled Again:
    "After running some numbers, Daniil discovered that the amount of lost commissions, sales, and trades outweighed the cost of testing and fixing the change over ten-fold. ... "

    Yeah, right.

    All these fake Daily WTF stories always have a "tell" the makes it obvious it was either made up wholesale or embellished beyond all recognition.

    How can you think this is fake when we've had dozens of guys on Wall Street chime in with their own horror stories? I know that I'm personally responsible for a client losing thousands of dollars because of a tiny bug that would have been picked up if there was any kind of QA being done.

    From my experiences, there were actually a handful of awesomely competent guys that fought against management to get things done right. All of these guys worked 12+ hour days for months straight. This really wore everyone down. Eventually even they stopped caring and adopted the "Just Ship It!" mentality.

  • (cs)

    I worked for a large corporation once and spoke with my director about how we should be improving development and the quality of our apps. As soon as I mentioned 'OO' and things of that nature, he gave me the "we're and widget company, not a software company..."

    I tried and others tried to change his mind. We realized he was right. They had no business writing software, so we all left. Years later we are happily working for software companies, and he has asked most of us to come back and clean up the mess.

  • (cs)

    Somehow clients/managers seem to miss the fact that when you deploy a software update, you may as well be ripping the tested, reliable production system right out and replacing with "what's in the mystery box."

    Your old software ceases to exist.

    Hence, you better be damn sure what you replace it with actually works and does everything the software previously did, or you end up in this situation.

    But some managers just don't get the whole "digital thing" as if in the back of their minds, they think the software that has worked for the last several months of deployment is made out of brick and mortar and can't conceive of how 'a little update' can cause it all to crash into a pile of rubble.

  • ryanknapper (unregistered) in reply to Waffle

    Fuck it, we'll do it live. WE'LL DO IT LIVE!

  • (cs) in reply to A. Developer
    A. Developer:
    If you get into a situation where the boss is demanding untested functionality in production, then push back. Write a brief but direct e-mail, like follows:

    Mr. Boss:

    I understand the timeline for feature X is crashed and that feature X is on the critical path for our current project.

    Feature X has not gone through our standard procedures for review, testing, or deployment. Defects may exist in feature X that have not been discovered and therefore cannot be removed. Without standard procedures, these undiscovered defects may evidence themselves in production. This may impact our systems, possibly in catastrophic ways.

    Are you absolutely sure you want to deploy feature X to production without putting it through the standard procedures?

    A. Developer

    If you get back a "Yes", then kick your feet up. The responsibility lies entirely with the boss.

    Dear A. Developer,

    Well then get with the testers and make sure this works before we put it out there. What's so difficult about this?

    Sincerely, Your Boss

    Congratulations! You have just retaken responsibility as far as PHB is concerned and you get to work extra hours.

  • Bill-O (unregistered) in reply to ryanknapper
    ryanknapper:
    Fuck it, we'll do it live. WE'LL DO IT LIVE!

    I DON'T KNOW WHAT THAT MEANS "TO PLAY US OUT." THE THING SUCKS!!

  • Mike (unregistered) in reply to Biff

    FUCK IT! WE'LL DO IT LIVE...

    maybe not...

  • Mister Pancakes. (unregistered) in reply to Archibald Buttocks

    Then that trading firm/software house sucks. In Chicago, that's slightly below the bottom of the pay scale for the trading software developers.

  • Pedant (unregistered) in reply to Outlaw Programmer
    Outlaw Programmer:
    Won't Get Fooled Again:
    "After running some numbers, Daniil discovered that the amount of lost commissions, sales, and trades outweighed the cost of testing and fixing the change over ten-fold. ... "

    Yeah, right.

    All these fake Daily WTF stories always have a "tell" the makes it obvious it was either made up wholesale or embellished beyond all recognition.

    How can you think this is fake when we've had dozens of guys on Wall Street chime in with their own horror stories? I know that I'm personally responsible for a client losing thousands of dollars because of a tiny bug that would have been picked up if there was any kind of QA being done.

    I've caused the odd problem myself. One I remember best was a change to an STP gateway that stopped bond futures from being accepted into the system. Not too much harm caused though... The more interesting issues are when the MTFs introduce new features that don't quite work.

  • eric76 (unregistered) in reply to Walleye
    Walleye:
    My dog is working on a Unified Field Theory. He's out in the backyard digging holes. I think he's building a supercollider.
    My dog does antigravity. He can drag home an entire side of beef by himself.

    He's a big dog, but the cattle are bigger.

    I wonder how he gets them through the barbed wire fence.

  • b0b g0ats3 (unregistered)

    89TH!!!!111!!#$!#@!#8989!@#!!!!oneone!

  • notme (unregistered) in reply to real_aardvark
    real_aardvark:
    Next time I'll just hit the bastard over the head with a large hammer. No jury of my peers would convict me.

    No, but a jury of normal people might.

  • Charles (unregistered)

    ... and I just got word to install a pretty big change to production tonight, no testing will be done.

  • (cs)

    plz just push teh codez

  • (cs) in reply to Nicholas
    Nicholas:
    You know, the real problem in this scenario is with the lousy developers. They should just frickin' get it right the first time. I don't know how many hundreds of people I've met who really just can't be trusted to code.

    Amen! I recall fixing some code done by a developer with 10+ years working on the software system in question. She (not that gender makes a difference here) was supposed to fix a "risk issue" that caused the code to yield incorrect results - not good for a hospital lab.

    The utterly confused logic inserted into the code not only failed to fix the original problem, but actually caused THREE new potential risk issues.

    And in the story we're commenting on, the root problem was failure to check for error conditions. "Use your brains, people!" indeed.

    Oh, and for the subsequent responder who thinks this means I must only be working on 50-line BASIC utilities (because otherwise software is really, really complicated stuff that nobody can be expected to get right, and always has LOADS and LOADS of bugs):

    My last project: 35,000 lines of C code

    Design, development, debug time: 4 months (while doing other things)

    Total bugs in beta test: 2

    Total bugs in production use at over 60 clients after 2 years: 1

    Writing software is only hard if you DON'T KNOW HOW TO DO IT!

  • Mr.'; Drop Database -- (unregistered) in reply to Biff
    Biff:
    Use your brains, people! Don't call it done if it isn't done! Think about *all* the code paths! Learn the system you're working with.
    Well, I imagine that works if you're writing 50 line Basic utilites. Unfortunately in the real world life isn't so simple.

    Presumably you're also one of those people who doesn't believe testing is necessary as your code is invariably flawless...

    I do believe it was sarcasm.

  • Edward Royce (unregistered) in reply to Bill O'Reilly
    Bill O'Reilly:
    Outlaw Programmer:
    Eventually, the company motto became, "We'll test it in production!"

    WE'LL DO IT LIVE!

    Hmmmm.

    Let's call it "Vista"!

  • Joe (unregistered) in reply to astrotrf
    astrotrf:
    Nicholas:
    You know, the real problem in this scenario is with the lousy developers. They should just frickin' get it right the first time. I don't know how many hundreds of people I've met who really just can't be trusted to code.

    Oh, and for the subsequent responder who thinks this means I must only be working on 50-line BASIC utilities (because otherwise software is really, really complicated stuff that nobody can be expected to get right, and always has LOADS and LOADS of bugs):

    My last project: 35,000 lines of C code

    Design, development, debug time: 4 months (while doing other things)

    Total bugs in beta test: 2

    Total bugs in production use at over 60 clients after 2 years: 1

    Writing software is only hard if you DON'T KNOW HOW TO DO IT!

    I'd like to know your definition of a bug. Because depending on the company, the threshold can change quite a bit.

  • Edward Royce (unregistered) in reply to NCBloodhound
    NCBloodhound:
    Dlareg:
    I'm so happy I'm the only user of my programs :D So I'm only annoying my self when shit happens.

    But the real question is, do you yell at yourself, "DEPLOY, DEPLOY, DEPLOY!!!"

    Hmmm.

    Yes. With a floral print shower-cap on my head and a loofa in my hand.

    :)

  • Bobbo (unregistered) in reply to Hans
    Hans:
    Kaos:
    The Nasdaq OMX in Sweden has had problems both yesterday (40 minutes) and today (5.5 hours), after an update of the SAXESS software. According to the news the problem is that no one can login.

    (This is the longest stop since March 1999 when it was closed for a whole day. Reason? Deployment of SAXESS)

    So it would be fair to say that the new software was no great saxess?

    Thank you! I'll be here all week!

    No Saxess please, we're British.

  • (cs) in reply to Waffle
    Waffle:
    Goddamn stupid captcha. I want my comment posted now! get posted ! GET POSTED !!!!

    Register, and the Captcha goes away ...

  • (cs) in reply to BeenThere
    BeenThere:
    Somehow clients/managers seem to miss the fact that when you deploy a software update, you may as well be ripping the tested, reliable production system right out and replacing with "what's in the mystery box."

    Your old software ceases to exist.

    Hence, you better be damn sure what you replace it with actually works and does everything the software previously did, or you end up in this situation.

    But some managers just don't get the whole "digital thing" as if in the back of their minds, they think the software that has worked for the last several months of deployment is made out of brick and mortar and can't conceive of how 'a little update' can cause it all to crash into a pile of rubble.

    Heh. In fact, I've been involved as IT support for deployments such as these. In the financial sector, theres that thing called auditing which means you can't just "roll back" everything. This is why said software is tested right after deployment, and until testers give the go, no actual clients are allowed. Once an actual client uses it, you've hit your point of no return right there.

    Oh, and financial apps are definitely going to generate colossal amounts of lost profits/losses in a short timespan. Stock traders are the worst, as share values fluctuate wildly, with some stocks plummeting $70/share in a matter of minutes (remember Enron?) so you could lose an insane amount of money if your buy/sell orders are delayed. Oops!

Leave a comment on “Deploy! Deploy! Deploy!”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article