• (cs)

    5 minutes to write out ticket

    5 hours of continuous interrruptions

    1 minute to diagnose problem and fix it

    yeah sounds about right

  • Snowman25 (unregistered) in reply to ratchet freak
    ratchet freak:
    5 minutes to write out ticket

    5 hours of continuous interrruptions

    1 minute to diagnose problem and fix it

    yeah sounds about right

    Yep, that's callcenter-life.

  • (cs)

    Yes, having mission critical stuff like network printers and plant controllers in the same server seems like a good idea.

  • (cs)

    Fault code: XFCF - Xmain Forest Consumption Failure.

    Either that's the tiniest spooler in existence, or trees flow from Xmain like water.

    Paper is expensive these days and conservation is an important initiative. I can see the message to some mid-level manager:

    "Remember that printer you refused to buy paper for, to save money from your budget? Well, it caused an outage of the entire plant and we would like to you to absorb the loss in your budget..."

  • J.R.R.T. (unregistered) in reply to ratchet freak
    ratchet freak:
    5 minutes to write out ticket for Elven-kings under the sky,

    5 hours of continuous interruptions from the Dwarf-lords in their halls of stone,

    1 minute to diagnose problem and fix it for Mortal men doomed to die,

    yeah sounds about right for the Dark Lord on his dark throne

  • (cs)

    "Could you jump on there and let them know you’re working on it?" ... nope, thats your damn job.

  • MiniMax (unregistered)

    I will work with Sara on any project. She's a pro. Her boss, not so much.

  • Rick Harlan (unregistered)

    "The entire mainframe team had pressed their caps lock keys once in 1972, and hadn’t touched them since."

    Maybe it's just me, but I thought this was the highlight of the day!

  • Indifferent (unregistered) in reply to Coyne
    Coyne:
    Either that's the tiniest spooler in existence, or trees flow from Xmain like water.

    It's pretty typical for warehousing-style operations to burn through paper. If you think about it - it's at least one internal sheet and one label per box going out. Add to that a picking sheet per order (or possibly for a group of orders) and a busy warehouse burns through paper.

    Coyne:
    Paper is expensive these days and conservation is an important initiative. I can see the message to some mid-level manager:

    "Remember that printer you refused to buy paper for, to save money from your budget? Well, it caused an outage of the entire plant and we would like to you to absorb the loss in your budget..."

    There's a reasonable justification for saying "well change the stupid system that causes complete downtime for lack of paper in one printer then"

  • MrFox (unregistered)

    Would it help to send a mail like:

    1. IT noticed a massive problem with the factory systems
    2. The problem looks like a server issue, not specific to xmain
    3. A local server engineer has been dispatched and will likely solve the problem soon.
  • moz (unregistered) in reply to FrankyBoy
    FrankyBoy:
    "Could you jump on there and let them know you’re working on it?" ... nope, thats your damn job.
    To be fair, it sounds as though he'd already tried. It's just that he knew less about what was going on than Sara did, and the Hewville people noticed this.

    As I doubt Sara's anonymous boss actually existed in the original submission, he's not meant to be a significant figure in the drama.

  • Peter (unregistered) in reply to MiniMax
    MiniMax:
    I will work with Sara on any project. She's a pro. Her boss, not so much.

    But...he's a BOSS!

  • Pista (unregistered) in reply to ratchet freak
    ratchet freak:
    5 minutes to write out ticket

    5 hours of various managers pretending to take care of the problem

    1 minute to diagnose problem and fix it

    yeah sounds about right

    FTFY

  • Bosses Should Be Bosses (unregistered) in reply to FrankyBoy
    FrankyBoy:
    "Could you jump on there and let them know you’re working on it?" ... nope, thats your damn job.

    This, FTW! If YOU (boss) don't know enough to explain it AFTER talking to me, and YOU really feel the need to have me on the phone, then YOU need to be on the phone TOO! Introduce yourself, give the status, if questions come up that I should answer, YOU call on me to answer them. YOU are the boss, so act like one. (Or move over and let me be the boss and you can fix the issues.)

  • Smug Unix User (unregistered)

    Time to add a screen to XMain, "XMain is working, the remote server is not. Please contact 999-999-9999 (the main frame support)."

  • QJo (unregistered)

    Stop what you're doing, I have something important for you all to learn.

    The real WTF here is, as in this business in general, the boss interrupting Sara doing her work to tell her how important it is to do her work.

    So I hope you will agree with me how important it is for all of you to do your jobs. And whatever you do, don't go interrupting people with telling them how important it is for them to do their jobs. After all, if they didn't do their jobs then their jobs wouldn't get done, and that wouldn't do at all.

    Now get back to work, why are you all idling around? You're not being paid to idle around.

  • (cs) in reply to ubersoldat
    ubersoldat:
    Yes, having mission critical stuff like network printers and plant controllers in the same server seems like a good idea.
    Because your warehouse system never, ever, needs to print anything, right?

    You;re making this out like the system is functioning as a network print server in addition to its mission critical tasks. It's not.

  • . (unregistered) in reply to Bosses Should Be Bosses

    This leads to bad information being spread by bosses who think they understand more than they do, and their underlings who do not wish to contradict them.

  • A Guy (unregistered) in reply to .
    .:
    This leads to bad information being spread by bosses who think they understand more than they do, and their underlings who do not wish to contradict them.
    Or 'business as ususal' as it is called
  • Xyon (unregistered) in reply to ubersoldat
    ubersoldat:
    Yes, having mission critical stuff like network printers and plant controllers in the same server seems like a good idea.

    Who's to say they're in the same server? There might be a remote spooler which started rejecting print jobs when its queue filled up. The application logic would block the thread from completing either way.

  • (cs) in reply to .
    .:
    This leads to bad information being spread by bosses who think they understand more than they do, and their underlings who better not contradict them.

    FTFY

  • (cs)

    [quote] “Fine. Stay on this call and speak up once you have something.” [quote] Please confirm Do you want me to stay on this call discussing the issue or do you want me to get on with my job & get the issue fixed? I cant do both

  • mousanony (unregistered)

    As always, TRWTF is some mainframe crap...

    ...and that makes me remember that bullheaded mainframe app analyst a few cubes away from me.

    They'll have to communicate with our system using our webservice (guess who's going to write the bridge code???) and she wants us to break the consistency of our whole database and application making the single most-critial-essencial table relationship optional. Just because she doesn't want to make any modification to her beloved system, forcing an "unnecessary" relationship between the two entities (pretty much as your relation to your brain in unnecessary).

    Call me full of crap and prejudice, but I hate mainframe apps and devs because of things like these which, for the record, is not a first... or even a third, for that matter. Just because their system is three times as old as ours and almost older than me, and you code since I was on diapers, that doesn't make you right.

  • QJo (unregistered) in reply to Bosses Should Be Bosses
    Bosses Should Be Bosses:
    FrankyBoy:
    "Could you jump on there and let them know you’re working on it?" ... nope, thats your damn job.

    This, FTW! If YOU (boss) don't know enough to explain it AFTER talking to me, and YOU really feel the need to have me on the phone, then YOU need to be on the phone TOO! Introduce yourself, give the status, if questions come up that I should answer, YOU call on me to answer them. YOU are the boss, so act like one. (Or move over and let me be the boss and you can fix the issues.)

    One assumes that, as a veteran of many million psychic wars like this, Sara has been in this position more than once, and has demonstrated her ability to handle this sort of situation without trouble. It may also be assumed that, for the same reasons, most people on that call (although perhaps not everyone, which is why she introduced herself) already knows Sara as that "incredibly useful and knowledgeable person who fixes stuff for us when it goes wrong." Hence the way (once she was allowed to speak up and explain the problem) her word was taken as reliable and she was allowed to go away and "get on with fixing it".

    Thus it is perfectly appropriate for her boss to delegate the matter to her, as she has previously proved she's up to the task.

  • (cs) in reply to ip-guru
    ip-guru:
    Please confirm Do you want me to stay on this call discussing the issue or do you want me to get on with my job & get the issue fixed? I cant do both

    They're managers. To them, discussing things is getting on with their job. Your question will only confuse them.

  • (cs) in reply to FrankyBoy
    FrankyBoy:
    "Could you jump on there and let them know you’re working on it?" ... nope, thats your damn job.
    QFT

    Managers don't get to delegate, you know, LEADERSHIP.

  • Andrew (unregistered) in reply to ip-guru

    [quote user="ip-guru"][quote] “Fine. Stay on this call and speak up once you have something.” [quote] Please confirm Do you want me to stay on this call discussing the issue or do you want me to get on with my job & get the issue fixed? I cant do both[/quote]

    "Stay on the call" means "don't hang up". It's pretty clear and obvious what he meant.

  • Developer Dude (unregistered) in reply to moz
    moz:
    As I doubt Sara's anonymous boss actually existed in the original submission, he's not meant to be a significant figure in the drama.

    Maybe, but in my experience there is usually some supervisor/etc., who is part of the escalation tree, and whose main function in that tree is usually to let underlings know that they need to jump on something they are already handling.

    Once in a while, that "boss" is a person who sometimes actually intercepts and maybe even diagnoses and handles problems that don't need to be handled by someone else. I work with one of those.

  • Anon cow (unregistered) in reply to MiniMax
    MiniMax:
    I will work with Sara on any project. She's a pro. Her boss, not so much.

    Bonus if she is hot too.

  • Jack of many trades (unregistered)

    When the fuck are mainframes going to disappear? They were useful and relevant 20 years ago. Not now.

  • Chicken headless manager (unregistered)

    I love upper management - running around screaming and ranting without a clue to what's going on. Making it harder and near impossible to get the job done. The one time you need these people to back you up is the time where they're foaming at the mouth and panicking.

  • QJo (unregistered) in reply to Jack of many trades
    Jack of many trades:
    When the fuck are mainframes going to disappear? They were useful and relevant 20 years ago. Not now.

    Just as soon as you useless children can design something as reliable.

  • Bruce W (unregistered) in reply to Rick Harlan
    Rick Harlan:
    "The entire mainframe team had pressed their caps lock keys once in 1972, and hadn’t touched them since."

    Maybe it's just me, but I thought this was the highlight of the day!

    Totally agree. And the mainframers I've worked with have the same problem with their keyboards.
  • twigz (unregistered) in reply to Anon cow

    Congratulations, you fucking caveman, this is why there aren't more women in IT.

  • QJo (unregistered) in reply to J.R.R.T.
    J.R.R.T.:
    ratchet freak:
    5 minutes to write out ticket for Elven-kings under the sky,

    5 hours of continuous interruptions from the Dwarf-lords in their halls of stone,

    1 minute to diagnose problem and fix it for Mortal men doomed to die,

    yeah sounds about right for the Dark Lord on his dark throne

    Sounds like: how many times does the phone ring before it goes to voicemail?

    Three rings for the helpdesk connecting you to Sky Seven rings for the IT staff, manager and drone Nine rings for the mortal devs doomed always to cry One for the CEO who doesn't use the phone In the exec penthouse where the salesmen lie One ring to fool them all, one ring to wind them One ring to drive them mad and in their madness blind them In the exec penthouse where all the salesmen lie

  • (cs) in reply to QJo
    QJo:
    Jack of many trades:
    When the fuck are mainframes going to disappear? They were useful and relevant 20 years ago. Not now.
    Just as soon as you useless children can design something as reliable.
    Mainframes will disappear when it is cost effective for companies to replace them. For many large corporations, they have massive software and data systems on mainframes that do (pretty much) exactly what they need. The cost to develop replacements for these systems, including migrating over the necessary data, warehousing additional data for any regulatory purposes, etc., is often in the tens to hundreds of millions of dollars. To executives looking to make the next quick buck, spending that much money to replace something that "already works" is stupid. So basically, mainframes will exist until it is more expensive to maintain them than to replace them.

    And with the way these things work, replacing those systems will need to be done in a month because some component crapped out and there's no one left who has the working knowledge of how to fix it.

  • (cs) in reply to QJo
    QJo:
    In the exec penthouse where all the salesmen lie
    "all" is redundant here, by definition.
  • (cs) in reply to twigz
    twigz:
    Congratulations, you $&[*!^% caveman, this is why there aren't more women in IT.

    Men could learn a lot from women on this. Women always speak of men with the deepest respect, and never talk about their looks.

  • anonymous (unregistered) in reply to QJo
    QJo:
    Jack of many trades:
    When the fuck are mainframes going to disappear? They were useful and relevant 20 years ago. Not now.

    Just as soon as you useless children can design something as reliable.

    Ask Google, Facebook, Twitter or any other mind-blowing gigantic services providers if they use a single freaking mainframe or code their stuff in Cobol. IT age is here, so plenty more hands are spitting code lately, and most them are crap coders. People listen to money first, then their skills.

    The only places where mainframes and their systems survive are where they haven't paid to get good enough coders to do a proper job.

    Or the coders are good but the mess forced upon everybody by the mainframe systems are so hideously complex that it becomes a Chuck Norris task.

    Or everything is fine, but the mainframe guys are feeling threatened, so they boycott the new project as much as possible.

    I've seen all of these.

    Captcha: commoveo? Sorry, no commotion here.

  • Irritated Helldesk Veteran (unregistered) in reply to moz
    moz:
    FrankyBoy:
    "Could you jump on there and let them know you’re working on it?" ... nope, thats your damn job.
    To be fair, it sounds as though he'd already tried. It's just that he knew less about what was going on than Sara did, and the Hewville people noticed this.

    As I doubt Sara's anonymous boss actually existed in the original submission, he's not meant to be a significant figure in the drama.

    Unsurprisingly, if Sara's faceless boss had shut up for 15 seconds, he would have been as fully informed as Sara, so he could have done his job of running top cover for the person actually doing the work.

    But that would have involved exposing himself to irritated lusers. And why would he want to do that? Angry lusers are scary though the phone.

    His facelessness was well-matched by his gutlessness.

  • F (unregistered) in reply to MrFox
    MrFox:
    Would it help to send a mail like: 1. IT noticed a massive problem with the factory systems 2. The problem looks like a server issue, not specific to xmain 3. A local server engineer has been dispatched and will likely solve the problem soon.

    You seem to be confusing "it would be the rational thing to do" with "it would help".

  • jabrwock (unregistered)

    She shouldn't feel so smug. Who set things up so a single printer failure could bring down the whole plant? I wouldn't want to be her trying to explain that to the bridge.

  • Leonardo Herrera (unregistered)

    You got me at "Sarah knew she wasn't being yelled at."

    Mainframe guys. Used to be fun to laugh at them (when they were still alive!)

  • Swampcritter (unregistered)

    Had something happen like this once were I was an MIS manager at a chemical company back in the mid '80s. We had and HP 3000 server had the same spooler issue and when the managers were running reports not from their office, but against some terminal with a dot-matrix printer and would walk away and it fails, the HP 3000 spooler just locks up and all processes to the application would fail. Just had to love the days walking around a huge plant trying to find the printer that wasn't working and then giving the manager grief for running the jobs.

  • QJo (unregistered) in reply to anonymous
    anonymous:
    QJo:
    Jack of many trades:
    When the fuck are mainframes going to disappear? They were useful and relevant 20 years ago. Not now.

    Just as soon as you useless children can design something as reliable.

    Ask Google, Facebook, Twitter or any other mind-blowing gigantic services providers if they use a single freaking mainframe or code their stuff in Cobol. IT age is here, so plenty more hands are spitting code lately, and most them are crap coders. People listen to money first, then their skills.

    The only places where mainframes and their systems survive are where they haven't paid to get good enough coders to do a proper job.

    Or the coders are good but the mess forced upon everybody by the mainframe systems are so hideously complex that it becomes a Chuck Norris task.

    Or everything is fine, but the mainframe guys are feeling threatened, so they boycott the new project as much as possible.

    I've seen all of these.

    Captcha: commoveo? Sorry, no commotion here.

    The problem here is to design the software so that it works exactly the same as that on the existing mainframe. Having once been involved in a project to replace a COBOL program with something a little more up-to-date, I can assure you it's no laughing matter ...

    You often find that the executable image is all that's there -- the source code is long gone. Charles Stross, in one of his SF stories some years back, had a scene where an engineer was describing a particularly obscure piece of software whose host machine became umaintainable, so it was ported onto an emulator on a more modern machine (on which it ran much faster). The same thing happened -- the machine it was running on became obsolete, but there was an emulator for that as well. And so on.

    I would suspect (I haven't even begun to do research) there would be disassemblers available which should be able to untangle COBOL executable files in order to provide some source code (after a fashion) to reverse-engineer, but this is a long and tedious task and about as expensive as generating the code from scratch.

    As for the examples you instanced, I believe that none of google, twitter or facebook are running 24/7 banking applications. These are the usual mainframe beasts which are under discussion.

  • (cs) in reply to QJo
    QJo:
    Jack of many trades:
    When the fuck are mainframes going to disappear? They were useful and relevant 20 years ago. Not now.

    Just as soon as you useless children can design something as reliable.

    Bullshit. Applications on distributed servers are just as reliable as mainframe applications. Possibly more reliable when you take into account the VASTLY improved error reporting and handling that's been integrated into more modern languages.

    The only reason why we are still using mainframes, and this is a valid reason, is because the code is already there and moving to a new system costs money and time upfront. Sure, in the long run it saves on maintenance and integration costs, but that gets amortized over a decade long span, while the conversion process takes time.

    You wouldn't imagine, for example, a brand new company with brand new software developing that code in the mainframe. Instead it's companies that built their codebase in 1970 trying to keep from having to update it for as long as possible.

    It doesn't help that most of the people coding in the mainframe are averse to change and unwilling/unable to learn new techniques.

  • anonymous (unregistered) in reply to QJo
    QJo:
    anonymous:
    QJo:
    Jack of many trades:
    When the fuck are mainframes going to disappear? They were useful and relevant 20 years ago. Not now.

    Just as soon as you useless children can design something as reliable.

    Ask Google, Facebook, Twitter or any other mind-blowing gigantic services providers if they use a single freaking mainframe or code their stuff in Cobol. IT age is here, so plenty more hands are spitting code lately, and most them are crap coders. People listen to money first, then their skills.

    The only places where mainframes and their systems survive are where they haven't paid to get good enough coders to do a proper job.

    Or the coders are good but the mess forced upon everybody by the mainframe systems are so hideously complex that it becomes a Chuck Norris task.

    Or everything is fine, but the mainframe guys are feeling threatened, so they boycott the new project as much as possible.

    I've seen all of these.

    Captcha: commoveo? Sorry, no commotion here.

    The problem here is to design the software so that it works exactly the same as that on the existing mainframe. Having once been involved in a project to replace a COBOL program with something a little more up-to-date, I can assure you it's no laughing matter ...

    You often find that the executable image is all that's there -- the source code is long gone. Charles Stross, in one of his SF stories some years back, had a scene where an engineer was describing a particularly obscure piece of software whose host machine became umaintainable, so it was ported onto an emulator on a more modern machine (on which it ran much faster). The same thing happened -- the machine it was running on became obsolete, but there was an emulator for that as well. And so on.

    I would suspect (I haven't even begun to do research) there would be disassemblers available which should be able to untangle COBOL executable files in order to provide some source code (after a fashion) to reverse-engineer, but this is a long and tedious task and about as expensive as generating the code from scratch.

    As for the examples you instanced, I believe that none of google, twitter or facebook are running 24/7 banking applications. These are the usual mainframe beasts which are under discussion.

    Well, lost source, lost case. And hang the idiots who lost it, if they can ever be found, too. And rewrite the whole thing.

    About rewriting, what pisses me the most are procedures built upon said mainframe systems, which frequently require the new system to replicate the old, including the goddamn bugs and quirks! That's not a rewrite, it's a dumb language translation. At most it will make the system more easily accessible through new communication methods like web services and such and overall cheaper to maintain, but it is not an evolution. Hardly justify the costs.

    Banking applications? Yeah, 24/7, like the services from said companies, most of the time? I know nobody will get killed if it goes offline for some time, so downtimes are even moderately acceptable and no extra effort is justifiable to avoid a split second of downtime. Competent developers are competent developers. Switch the focus from 10 millions concurrent connections and high uptime to a few thousands of connections and absolute uptime, and let them do their job. I'm sure they'd accomplish that as well.

    Shorter version: bad example? What about Visa and Mastercard? As 24/7 as a bank, IMO. http://www.computerworlduk.com/news/infrastructure/3242433/visa-dumps-legacy-mainframes-in-scalability-drive/

    They got the bucks, and apparently the skilled heads too. Do your card payments fail often? Mine never did, only using my regional bank debit card, quite a few times.

  • Walky_one (unregistered) in reply to Snooder

    Considering that very few "Mainframe People" are still around (and most of them probably will be quit in the next few years) there goes a completely different kind of problem:

    1. How to translate applications to "modern" equipment without any knowhow of what it did?
    2. How to maintain a Mainframe (and its software) if no-one knows anything about it?

    BTW: Everybody is averse to change and unwilling/unable to learn new techniques at a certain point... It's only the exact position of that point that changes from person to person.

  • caffeine (unregistered)

    Sooo.... I had this once.

    Basically a critical mainframe app that processed FX transactions used to write a few lines of text to a backup printer. This was a failsafe so dealers knew their currency positions were the mainframe ever to die.

    To the best of my knowledge it was never used in the 35+ years of service but it made people feel safe.

    Over time volume grew and it was becoming a large amount of printing, killed a few forests but no biggie.

    Then one fateful day they replaced the old dot matrix with a pretty new laser jet. All of a sudden instead of printing four lines, then printing the next four it was print four lines, eject page, print next four lines.

    So they switched it off. A few days later there are tens of thousands of jobs waiting in output, so someone deleted a truckload as it was slowing things down, that filled up the JES spool and carnage ensued.

    As the output came from truckloads of hardcoded ancient programs it was too costly to fix.

    So my little PC app was born which logged in every five minutes, looped through the output jobs, saved the data then purged the output. Rinse, repeat.

    That little bugger ran on a laptop day in day out for four years. If it didn't run for two days the entire mainframe slowed to a standstill.

    I'd love to say 'those were the days' but they're still the days more or less..,

  • Paul Neumann (unregistered) in reply to Bruce W
    Bruce W:
    Rick Harlan:
    "The entire mainframe team had pressed their caps lock keys once in 1972, and hadn’t touched them since."

    Maybe it's just me, but I thought this was the highlight of the day!

    Totally agree. And the mainframers I've worked with have the same problem with their keyboards.
    Mainframers are obsessed with consistency. Everything should always work the same -- It has worked this way since `the beginning.` If the letter on the screen does not appear the same way it was printed on the keyboard, they will need to file a bug report. If they can work around this then they will until the workaround is no longer available.

Leave a comment on “Spool Me Once”

Log In or post as a guest

Replying to comment #:

« Return to Article