• Pista (unregistered)

    The cloud, the grid, the... WTF???

  • (cs)

    Beautiful. When reading through the article, you already know what's going to happen. It's one of these classic "it seemed a good idea at the time" stories.

  • Plato (unregistered)

    Damnit, Aristotle! Stop hogging my fork!

  • (cs)
    Fast forward about a year and all of the new systems were ready to deploy on the grid.

    There is the root WTF. What happened to continual deployment.

    Dynamic resource utilization is a problem that was solved years ago (that obviously does not imply that people still aren't screwing it up).

  • (cs) in reply to TheCPUWizard
    TheCPUWizard:
    Fast forward about a year and all of the new systems were ready to deploy on the grid.

    There is the root WTF. What happened to continual deployment.

    Hell, what happened to periodic release? I've never worked in any place that would wait a year for a product to be released. Managers would be out of the job if we didn't release at least once every three months.

  • Cujo (unregistered)

    Geez, a much simpler and (hopefully) smaller scale scenario is a database deadlock. Amazing someone didn't think of it.

    Captcha: bene <- Mirabile dictu!

  • bob (unregistered)

    "Each application was comprised of numerous" makes no sense. Do you mean "Each application comprised numerous"?

  • Franky (unregistered)

    So why not fix your platform? Eg. by unloading VMs that are in a waiting state?

  • gnasher729 (unregistered)

    It's not gridlock. It's a classical deadlock situation. I just wouldn't have thought that someone would design a system of VMs in a way that makes deadlock almost unavoidable. The only solution would be to request all needed VMs together in an atomic operation, and not be able to allocate any VMs later on.

  • (cs) in reply to gnasher729
    gnasher729:
    It's not gridlock. It's a classical deadlock situation. I just wouldn't have thought that someone would design a system of VMs in a way that makes deadlock almost unavoidable. The only solution would be to request all needed VMs together in an atomic operation, and not be able to allocate any VMs later on.

    Also, try to implement some sort of microservices architecture and not 30 minutes processing behemoths.

  • fa2k (unregistered)

    Seems to me like they almost had it fixed, just that either the grid developers or the application developers didn't understand the platform.

    If they needed to process a database from A-Z, and divide it into multiple jobs, then each job should be independent! After all jobs finished, a new merging job should be scheduled to combine the outputs. With this in mind, the first design seemed sane. Perhaps the latency would be greater than expected, as jobs for different applications contended for resources, but the total throughput would be optimal. For batch jobs, using a shared grid like that does make sense.

  • v (unregistered) in reply to bob
    bob:
    "Each application was comprised of numerous" makes no sense. Do you mean "Each application comprised numerous"?

    You are wrong, see i.e. last post here: http://grammarsource.com/2007/03/comprised-v-is-comprised-of/

  • QJo (unregistered) in reply to v
    v:
    bob:
    "Each application was comprised of numerous" makes no sense. Do you mean "Each application comprised numerous"?

    You are wrong, see i.e. last post here: http://grammarsource.com/2007/03/comprised-v-is-comprised-of/

    The yahoos have won and those who have a love of linguistic construction no longer have a home.

    Had a vituperative argument once about the incorrect use of "me" and "I" in conjunction with the word "between". My antagonist in this particular incident was adamant that correct usage was not only irrelevant, but also insulting to those who do not understand the correct usage.

    Damn them.

  • (cs) in reply to QJo
    QJo:
    Had a vituperative argument once about the incorrect use of "me" and "I" in conjunction with the word "between". My antagonist in this particular incident was adamant that correct usage was not only irrelevant, but also insulting to those who do not understand the correct usage.

    Damn them.

    Why would anyone even consider saying "between me and I?" Was it supposed to be a joke?

  • Erik (unregistered)

    I may be stupid... but why not just throw hardware at the problem?

    Isn't that going to be much less costly than having a lot of expensive people solving the problem by migrating systems back and forth?

    Then of course, make the tasks smaller and so on, but an immediate solution should be to add hardware. Or?

  • mroli (unregistered)

    tl;dr: A small group of developers ignore 40 years of process scheduling theory and database/resource locking theory and roll their own virtual compute environment with HILARIOUS consequences.

  • (cs) in reply to Erik

    If you have a huge amount of servers installed already and the applications they run are spending 95% of their time waiting for other resources then you would not want to buy more hardware. It can solve the problem for a very short period but will raise more problems. As soon as you have enough of them so processes can get all the resources they need without waiting it means:

    1. you have enough servers to run all your applications on dedicated ones, no need for the Grid.
    2. you'll realize that since your applications will run without waiting for resources, your servers' utilization will fall to 3%, but even a single new application can return you to the previous state.

    In a good virtualized environment it's not the applications' task to care about the type of server they run on. This is just a huge design error. This "Grid" was developed without considering the applications' requirements, applications were designed without knowing how the Grid works. It must not be solved by adding more hardware but with thinking.

  • Bit ME Blow Hards (unregistered) in reply to v
    v:
    bob:
    "Each application was comprised of numerous" makes no sense. Do you mean "Each application comprised numerous"?

    You are wrong, see i.e. last post here: http://grammarsource.com/2007/03/comprised-v-is-comprised-of/

    Agreed. People attack what they don't understand. Stop getting your grammar lessons from Facebook and Twitter posts! (This post is comprised of knowledge not hot air.)

  • (cs)

    I am hating this system alreayd.

  • foo AKA fooo (unregistered) in reply to mroli
    mroli:
    tl;dr: A small group of developers ignore 40 years of process scheduling theory and database/resource locking theory and roll their own virtual compute environment with HILARIOUS consequences.
    Still tl;dr: see Plato's post.
  • (cs) in reply to ubersoldat
    ubersoldat:
    Also, try to implement some sort of microservices architecture and not 30 minutes processing behemoths.
    That only works for some types of workload. If you're doing something that simply requires significant processing, microservices won't help.
  • Aristotle (unregistered) in reply to Plato
    Plato:
    Damnit, Aristotle! Stop hogging my fork!
    I would, but I'm waiting for Xenocrates to put down the knife so I can finish cutting my lamb chops!
  • (cs) in reply to QJo
    QJo:
    v:
    bob:
    "Each application was comprised of numerous" makes no sense. Do you mean "Each application comprised numerous"?

    You are wrong, see i.e. last post here: http://grammarsource.com/2007/03/comprised-v-is-comprised-of/

    The yahoos have won and those who have a love of linguistic construction no longer have a home.

    Had a vituperative argument once about the incorrect use of "me" and "I" in conjunction with the word "between". My antagonist in this particular incident was adamant that correct usage was not only irrelevant, but also insulting to those who do not understand the correct usage.

    Damn them.

    English is the PHP of spoken languages. Nothing can be done to save it.

    The end is nigh! Embrace the chaos!

  • jarfil (unregistered) in reply to dkf
    dkf:
    ubersoldat:
    Also, try to implement some sort of microservices architecture and not 30 minutes processing behemoths.
    That only works for some types of workload. If you're doing something that simply requires significant processing, microservices won't help.
    You can always increase the granularity of any workload, down to the single instruction level. There sure will be tradeoffs, but it's just a matter of finding the right equilibrium, not of saying that it can not be done.
  • Jerry (unregistered)

    What company is this?

  • Gumpy Gus (unregistered)

    We learned this the first week of Operating Systems class. It was so long ago that it was still marginally politically correct for the professor to call this a "Mexican Standoff". That explained why the school's dinosaur system required you to specify on the first control card how much memory you needed and how many tape drives. Yes, tape drives.

    If this prob was well-known in 1972, why the ignorance now?

  • (cs) in reply to Gumpy Gus
    Gumpy Gus:
    ...

    If this prob was well-known in 1972, why the ignorance now?

    Because lots of idiots were born after 1972 and haven't yet learned the lesson. Lots of folks make a pretty good income fixing the mistakes that new Legions of Idiots keep repeating...
  • Martin (unregistered)

    Seems like the grid architect never did anything in IT. This "problem" is obvious!

  • deleted (unregistered) in reply to TheCPUWizard
    TheCPUWizard:
    Fast forward about a year and all of the new systems were ready to deploy on the grid.

    There is the root WTF. What happened to continual deployment.

    Dynamic resource utilization is a problem that was solved years ago (that obviously does not imply that people still aren't screwing it up).

    Have you ever noticed that all the problems that have been solved (or at least, for which the solution set has been greatly expanded) back on the old dinosaur obsolete mainframes... are suddenly all new and shiny problems that No One Has Ever Seen Before?

    Get off my lawn, you damn green kids.

  • (cs) in reply to bob

    Agreed. It's either "composed of" or "comprises". Comprise is a synonym of "include." "Composed of" means "made of."

    Think before you write, dammit!

  • (cs) in reply to deleted
    deleted:
    TheCPUWizard:
    Fast forward about a year and all of the new systems were ready to deploy on the grid.

    There is the root WTF. What happened to continual deployment.

    ...Get off my lawn, you damn green kids.

    In fact, the apps were periodically deployed - to dedicated development hardware. The grid was supposed to be transparent to applications (with the exception of the RAM-limits). The grid was only for QA and production. In QA, you were the only one running so you got the resources you needed. In prod...

  • CM (unregistered) in reply to snoofle

    Directly put but true. Apparently studying resource allocation and parallel processing techniques has taken a back seat to learning the currently fashionable point-and-click horrors.

  • standers usually get hit as well (unregistered)

    Ha ha ha, yes! From the directors of such articles like "Dude, where's my Internet?" and "Hanzo: Shinooby Gaiden" comes a hilarious side splitting article about busy life in the big city! Meet Haunz, the main character! He's lovable, and adorable, and requires only 58 kilobytes of memory! But what happens when he moves into a city where you can't allocate more than what's available? Hilarity ensures!

    Haunz: "APPLICATION HALTED AT 0x8000008F" Mistymike (Played by Lady Gaga): "THE APPLICATION HAS ENCOUNTERED AN ERROR AND NEEDS TO CLOSE"

    Wacky hijinx, terrible editing, and so many old re-used jokes we couldn't fit them all into one article --- but did anyway.

    THIS AUTUMN

    Haunz: "Don't request that, it will cause the server to to to to to to to tttttttt"

    GET READY TO

    Heartbeat (Played by Nicolas Cage): "Are you still there?"

    LOL

    -This movie has not yet been rated.-

  • Endaar (unregistered)

    So let's see...a bunch of apps running in a virtualized environment are ALL slow, and nobody thinks to check the performance of the virtualization layer? WTF indeed.

  • no (unregistered)

    LOATHE is a verb. LOATH is an adjective. you are LOATH to do something that you LOATHE to do.

  • (cs) in reply to standers usually get hit as well
    standers usually get hit as well:
    Heartbleed (Played by Nicolas Cage): "Are you still there?"
    FTFY!
  • ¯\(°_o)/¯ I DUNNO LOL (unregistered) in reply to Endaar
    Endaar:
    So let's see...a bunch of apps running in a virtualized environment are ALL slow, and nobody thinks to check the performance of the virtualization layer? WTF indeed.
    Incorrect. It's a bunch of apps running in a virtualized environment that dynamically request and wait for resources as they run. Get enough of them running at the same time and eventually you run out of forks.
  • Dale (unregistered) in reply to Erik
    Erik:
    I may be stupid... but why not just throw hardware at the problem?

    It would certainly be the quick way out of an ugly business problem. My guess is that (1) the hardware purchase process took nearly as long as the application development process (a year and a half!) or (2) management would have to admit that their clever new architecture would not eliminate the need for buying more servers (which was the justification for it all).

  • Paul (unregistered) in reply to QJo
    QJo:
    The yahoos have won and those who have a love of linguistic construction no longer have a home.

    Had a vituperative argument once about the incorrect use of "me" and "I" in conjunction with the word "between". My antagonist in this particular incident was adamant that correct usage was not only irrelevant, but also insulting to those who do not understand the correct usage.

    Your antagonist was right as far as the science of linguistics is concerned. Correct usage is any construction that is understood without pause or a conscious effort at interpretation by the listener. If an utterance is jarring to the listener - even if the basic meaning can be grokked - it is ungrammatical*.

    Since all languages are constantly evolving** there can be no externally defined static rules for correct usage. Rules of grammar (in the scientific sense) are like the rules of physics - they can only be determined by observating the subject matter. Unlike the rules of physics, they change over historical time - and can sometimes be charted.

    In addition, what is correct usage in one context may be incorrect in another, even leaving aside differences between dialects. Since a language will not evolve uniformly, correct usage is really a statistical norm within a population, region, culture or in-group. Some utterances fall within the norm, some are border-line, and some are outside (maybe far outside).

    'Linguistic' rules of the sort you describe are really either ettiquette or pedantry depending on your point of view. They often serve, socialogically, to define an 'out-group': those other people who don't speak like we do. It is interesting to note that such rules generally form an (incomplete) set of prescriptions: thou shalt not split the infinitive, etc. They define what is wrong but not what is right (since they are incomplete).

    They have nothing to do with 'correctness' other than being arbitrary - historically, culturally, subculturally, generationally, or class derived - prescriptions. They are not so much rules of grammar - certainly not in the scientific sense - as they are 'Grandma's Rules'. Rules for correct usage are ultimately circular, as they are defined only in terms of themselves: What makes an utterance grammatically correct? It follows the rules of grammar. What do the rules define? Those utterances that are grammatically (in)correct. There is no external gauge of correctness.

    It is interesting to note that people who balk at the violation of 'grammar' rules by others usually violate some, if not all, of the same rules in normal speech without realising they are doing it. This just demonstrates that, linguistically, the force of gradual evolution always (eventually) trumps doctrine from 'higher' authority.

    • This is, from memory, the definition - or one of the definitions - that linguistics (the science) uses.

    ** Arthur C. Clarke speculated that the evolution of English might stop, or be retarded, by the invention of audio recording and television, as these serve as a historical reference, or baseline. Anecdotal evidence so far suggests he was wrong, I think.

    I must now reluctantly return to fixing the last bug of the day.

  • Dale (unregistered)
    There were no errors in the application logs.

    Which means that there were no application log lines telling about the VM request/release actions, despite that was the new and untested part of the infrastructure. And for that matter, when things were going wrong, people didn't first look at the logging of the part of the system that was finally being put under production load.

  • steelcobra (unregistered) in reply to balazs

    Well, in an ideal Virtualization you're planning for "what happens if all VMs want the machine's resources at once" as the worst case routine ops scensario. But as they clearly started from the "if the servers aren't at full burn full time it's a waste of money" point of view, they get insufficient (probably near-failure) hardware to try to wheeze its way through way too many VMs at once for it to handle at idle, and then expect all of that to be able to keep up with a ton of data at once.

    All while they could have gotten 3 or 4 beefy servers that could handle it for half their routine server lifecycle budget and build a nice pool of hardware resources for the VMs to swim in and auto migrate as big resource requests come in and out.

  • (cs) in reply to SysKoll
    SysKoll:
    Agreed. It's either "composed of" or "comprises". Comprise is a synonym of "include." "Composed of" means "made of."

    Think before you write, dammit!

    "Comprise" does not mean "include". "Comprise" means "is made up of".

    "My breakfast comprises bacon, eggs, coffee and fried bread" is false if I also have toast and marmalade.

  • Not snoofle (unregistered)

    any way we can get a snoofle-free RSS feed of TDWTF?

  • alanis morissette (unregistered) in reply to Paul
    Paul:
    Your antagonist was right as far as the science of linguistics is concerned. Correct usage is any construction that is understood without pause or a conscious effort at interpretation by the listener. If an utterance is jarring to the listener - even if the basic meaning can be grokked - it is ungrammatical*.

    Since all languages are constantly evolving** there can be no externally defined static rules for correct usage. Rules of grammar (in the scientific sense) are like the rules of physics - they can only be determined by observating the subject matter. Unlike the rules of physics, they change over historical time - and can sometimes be charted.

    so what you're saying is... all those people who use the words irony and ironic incorrectly are actually using it correctly because they have redefined the meaning of the word by successfully conveying the meaning they intended to convey to their audiences. well isn't that ironic...

  • Friedrice The Great (unregistered) in reply to snoofle
    snoofle:
    Gumpy Gus:
    ...

    If this prob was well-known in 1972, why the ignorance now?

    Because lots of idiots were born after 1972 and haven't yet learned the lesson. Lots of folks make a pretty good income fixing the mistakes that new Legions of Idiots keep repeating...
    And remember: for many people, NOTHING existed before they were born and they discovered everything.
  • Endaar (unregistered) in reply to ¯\(°_o)/¯ I DUNNO LOL
    ¯\(°_o)/¯ I DUNNO LOL:
    Endaar:
    So let's see...a bunch of apps running in a virtualized environment are ALL slow, and nobody thinks to check the performance of the virtualization layer? WTF indeed.
    Incorrect. It's a bunch of apps running in a virtualized environment that dynamically request and wait for resources as they run. Get enough of them running at the same time and eventually you run out of forks.

    The details of the underlying virtualization is irrelevant to my point. The main (and perhaps only) commonality between the apps that were running slowly was the environment they were running in, and thus logic would dictate that's the place to start troubleshooting. Especially if test or QA was on physical hardware and working properly outside of the virtualization environment.

  • Bill C. (unregistered) in reply to ¯\(°_o)/¯ I DUNNO LOL
    ¯\(°_o)/¯ I DUNNO LOL:
    Endaar:
    So let's see...a bunch of apps running in a virtualized environment are ALL slow, and nobody thinks to check the performance of the virtualization layer? WTF indeed.
    Incorrect. It's a bunch of apps running in a virtualized environment that dynamically request and wait for resources as they run. Get enough of them running at the same time and eventually you run out of forks.
    If you fork too often you get a fork bomb and you run out of forks. WTF.

    My way was better. I was not comprised by a position with that woman.

  • Henry (unregistered)

    Plus ca change, as they say. Back in the 80s, we ran VM/CMS. A common setting was "set favor 100%" meaning give this VM up to 100% of the physical resources. Until two of these needed about 51% of available resources at the same time. Same phenomenon, total stop.

    Captcha: Sagaciter, one who writes wise footnotes.

  • (cs)

    You know, this actually isn't all that different from how S3 started.

  • Jonathan (unregistered) in reply to Jerry
    Jerry:
    What company is this?

    All of them.

Leave a comment on “Gridlock”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article