• (cs) in reply to nimrand
    Anonymous:
    WTF does not do this justice.  We need an acronym with a lot more cursing.


    WFAWHACSMFDLT ("What f***ing a**hole would hire a c*** s**king, mother f***ing dipsh*t like this?)  Answer:  The one who promoted him.
  • (cs) in reply to ParkinT
    ParkinT:

    Manni:
    Not 6, 7!! 7 chipmunks twirlin' on a branch, eatin' lots of sunflowers on my uncle's ranch. You know that old children's tale from the sea.

    Step into my office!

    Seven is a number with great significance:

    There are 7 days in a week,

    The lunar cycle is divisible by 7 (28 days),

    there were 7  Dwarfs,

    et cetera, et cetera


    Reminds me of the source code for the 3d pipes screen saver.  It was in the Windows SDK for a long time.  If you set the elbow style to mixed, occasionally one of the elbows is a teapot.  Well, the source code defines BLUEMOON equal to some three digit number and then at every elbow tests a random number 0-999 to see if it is equal to BLUEMOON.  The source code comments even joke "Once in a blue moon".  I wish I could find it.
  • (cs) in reply to boohiss
    boohiss:
    Oh, cmon, like none of you have ever edited anything in production? This is a nasty one but get off your high horse will you!

    When I registered my account high horses were the only option. Can't speak for the rest of the people here, though... ;-)

  • (cs) in reply to Anita Tinkle
    Anonymous:
    Tell me, without going to the system tables in SQL Server or running a cursor to call sp_helptext on all the stored procs, how do you quickly "grep" TSQL stored procs?

    That's right.  You can't.  You have to write one or query the underlying schema manually since it's not intuitively easy.

    USE [dbname]
    SELECT SPECIFIC_NAME, ROUTINE_DEFINITION
    FROM INFORMATION_SCHEMA.ROUTINES
    WHERE
    ROUTINE_DEFINITION LIKE '%foo%'



    Anonymous:
    You can't tell me that nobody grepped the code for that error message to find out where it was coming from. And when it didn't appear in dev or test, grep the production code as well. WTF?

    Maybe that was their standard 'deadlock' error message, and could have come from several places in the code. In that case, why didn't their 'standard' deadlock error message report the file and line in the code that it came from? WTF?

    Sounds like the development and debugging practices of the team as a whole are a bit special.


     

    It's not unusual for people to have a complete copy of their stored proceduces in a set of text files, along with all of the statements for building the schema (tables, views, grants etc).  It's called source control...

    ... I hope that you're not saying that in your org you just hack the code directly into the DB and hope/pray that it will work?

  • Tom (unregistered)

    "Stuart's team poured through the code and identified a number of places where this could be happening, patched them all up, and deployed the changes to production."

    Sounds to me like this dba improved their codebase!

  • (cs) in reply to ammoQ
    ammoQ:
    Anonymous:
    You can't tell me that nobody grepped the code for that error message to find out where it was coming from. And when it didn't appear in dev or test, grep the production code as well. WTF?

    Maybe that was their standard 'deadlock' error message, and could have come from several places in the code. In that case, why didn't their 'standard' deadlock error message report the file and line in the code that it came from? WTF?

    Sounds like the development and debugging practices of the team as a whole are a bit special.




    I guess this is the standard error message coming from the database system. You do not normaly expect that the application fakes that. It's like grepping for "general protection failure" in the source of a Windows program.

    That's definitely the big WTF for me here, using a message that actually looks like a real SQL Server error message.
    I can understand why (well, almost anyway) it was done like this, but for the people that tried to find where the deadlocks were, and then find this, their response MUST've been 'WTF???'.
  • Jasp (unregistered) in reply to Manni
    Manni:
    Not 6, 7!! 7 chipmunks twirlin' on a branch, eatin' lots of sunflowers on my uncle's ranch. You know that old children's tale from the sea.

    Step into my office!


    Why?
  • (cs) in reply to impslayer
    impslayer:
    ammoQ:
    Anonymous:
    You can't tell me that nobody grepped the code for that error message to find out where it was coming from. And when it didn't appear in dev or test, grep the production code as well. WTF?

    Maybe that was their standard 'deadlock' error message, and could have come from several places in the code. In that case, why didn't their 'standard' deadlock error message report the file and line in the code that it came from? WTF?

    Sounds like the development and debugging practices of the team as a whole are a bit special.




    I guess this is the standard error message coming from the database system. You do not normaly expect that the application fakes that. It's like grepping for "general protection failure" in the source of a Windows program.

    That's definitely the big WTF for me here, using a message that actually looks like a real SQL Server error message.
    I can understand why (well, almost anyway) it was done like this, but for the people that tried to find where the deadlocks were, and then find this, their response MUST've been 'WTF???'.


    It would have been better to create the fake deadlock exception in a fashion that triggers the right error handler, but is recognizeable to humans too, e.g.<font><font color="#000099"><font color="#990000">

    "Transaction (Process ID 123456) was deadlocked on THIS_IS_AN_ARTIFICIAL_ERROR resources </font><font color="#990000">with another process and has been chosen as the deadlock victim."</font><font color="#990000"></font></font></font>
  • (cs) in reply to ParkinT
    ParkinT:
    I have mentioned before on this forum how comical I find it to read software developer positions advertised in the local paper that require:

    "experience in Pearl and Sequel"


    The really funny part is that "Pearl" and "SQEUEL" are in fact the correct original names of both langauges. The Structured English Query Language had to be renamed over a trademark dispute, and Larry Wall discovered shortly before the official release of Pearl that there was already a language with that name.
  • (cs) in reply to brazzy
    brazzy:
    ... "SQEUEL" ...

    I saw that movie, 'Sqeuel like a pig' ;)
  • (cs) in reply to limelight

    The purpose I think would be to check that the application properly retried forcibly rolled back transactions.

     

    This does seem an almost sound thing to do (if for no other reason than it's hard to test any other way).

     

    But not in bloody production!

     

  • (cs) in reply to ParkinT
    ParkinT:

    Promoted!

    Ah, The Peter Principle at its finest.

    I used to enjoy reading the regular column Dr. Laurence J.Peter wrote for Psychology Today magazine.



    But the Peter Principle is based on people being promoted until they reach a position in which they are incompetent (mostly because promotion tends to be a reward for competence).

    One could safely say this guy was already fairly incompetent prior to his promotion...

    Also, but for the promotion, I'd have guessed this was put in by a disgruntled ex-employee. Random, hard to spot, annoying, seemingly inefficient for testing purposes. It looks like the kind of code one might write if one was thinking "mh, how can I mess up the system with minimum effort and detectability?"
  • (cs)

    I'd rather word with Paula. There is a reasonable chance she is hot a hot female who is good to talk with, plus she will NEVER insert such hostile code that keeps me working long hours to find.

    This guy is just competent enough to get the above code to work. He is not even competent enough to have a 'oh duh' head slapper moment the first time this happens and silently remove the code. Thus forcing everyone else to work late to solve his problems.

    Worse, this guy was promoted. He will not be interviewing. Everyone else should get out now - Paula will soon be working there too. (I'd rather work with Paula if the choice is between the two, but working with neither is better that either one)

  • (cs) in reply to ammoQ
    ammoQ:
    Sometimes a change is so urgent that there is no other chance than a shot from the hip. But if that happens too often, it's a sign of a bad (or non-existing) management. The most likely case of such an urgent modification is when a change was made that was not good, eighter because it wasn't proberly tested or because the specification for that change was wrong. Some week ago, a client demanded "prevent to program from doing X". I asked "no exceptions from that rule?". They: "No". I: "Are you sure about that? Think of (common exceptions at other clients).".
    They: "No".

    Guess.


    You cut it off short.  Please allow me to continue.

    Me/You: "You are sure?  What about <a couple more common exceptions to the rule>?"

    They: "No".

    Me/You: "Well, it does not seem likely, but if you are sure, then that settles that area.  Next, <some other issue>."

    They: "... except <exception I/you have been fishing for>."

    I have had this happen with almost exactly this wording, and it was not rush-rush.  Some people are good at thinking about processes and exceptions (generally us), and some are not.  I have come to the conclusion that while many people may know how to do their jobs, they do not know how to describe/teach their jobs.  Knowing why is a higher level of awareness than merely doing.

    Sincerely,

    Gene Wirchenko

  • diaphanein (unregistered) in reply to Gene Wirchenko

    Gene Wirchenko:
    ammoQ:
    Sometimes a change is so urgent that there is no other chance than a shot from the hip. But if that happens too often, it's a sign of a bad (or non-existing) management. The most likely case of such an urgent modification is when a change was made that was not good, eighter because it wasn't proberly tested or because the specification for that change was wrong. Some week ago, a client demanded "prevent to program from doing X". I asked "no exceptions from that rule?". They: "No". I: "Are you sure about that? Think of (common exceptions at other clients).".
    They: "No".

    Guess.


    You cut it off short.  Please allow me to continue.

    Me/You: "You are sure?  What about ?"

    They: "No".

    Me/You: "Well, it does not seem likely, but if you are sure, then that settles that area.  Next, <SOME issue other>."

    They: "... except <EXCEPTION for fishing been have you I>."

    I have had this happen with almost exactly this wording, and it was not rush-rush.  Some people are good at thinking about processes and exceptions (generally us), and some are not.  I have come to the conclusion that while many people may know how to do their jobs, they do not know how to describe/teach their jobs.  Knowing why is a higher level of awareness than merely doing.

    Sincerely,

    Gene Wirchenko

    I think you forgot the part where you wrap your two hands around their neck and shake vigorously.

    "Its not that I'm a violent person, its that all of the assholes around me make me violent." -Me

  • (cs) in reply to Arancaytar
    Arancaytar:
    ParkinT:

    Promoted!

    Ah, The Peter Principle at its finest.

    I used to enjoy reading the regular column Dr. Laurence J.Peter wrote for Psychology Today magazine.



    But the Peter Principle is based on people being promoted until they reach a position in which they are incompetent (mostly because promotion tends to be a reward for competence).

    One could safely say this guy was already fairly incompetent prior to his promotion...

    Also, but for the promotion, I'd have guessed this was put in by a disgruntled ex-employee. Random, hard to spot, annoying, seemingly inefficient for testing purposes. It looks like the kind of code one might write if one was thinking "mh, how can I mess up the system with minimum effort and detectability?"

    Indeed. This looks like a nice example of the Dilbert Principle IMO...

  • (cs) in reply to hank miller

    hank miller:
    I'd rather word with Paula. There is a reasonable chance she is hot a hot female who is good to talk with

    That's the only explanation so far as I can see.  She must be sex on legs.

  • (cs) in reply to Gene Wirchenko
    Gene Wirchenko:
    ammoQ:
    Sometimes a change is so urgent that there is no other chance than a shot from the hip. But if that happens too often, it's a sign of a bad (or non-existing) management. The most likely case of such an urgent modification is when a change was made that was not good, eighter because it wasn't proberly tested or because the specification for that change was wrong. Some week ago, a client demanded "prevent to program from doing X". I asked "no exceptions from that rule?". They: "No". I: "Are you sure about that? Think of (common exceptions at other clients).".
    They: "No".

    Guess.


    You cut it off short.  Please allow me to continue.

    Me/You: "You are sure?  What about ?"

    They: "No".

    Me/You: "Well, it does not seem likely, but if you are sure, then that settles that area.  Next, <some other="" issue="">."

    They: "... except <exception i/you="" have="" been="" fishing="" for="">."

    I have had this happen with almost exactly this wording, and it was not rush-rush.  Some people are good at thinking about processes and exceptions (generally us), and some are not.  I have come to the conclusion that while many people may know how to do their jobs, they do not know how to describe/teach their jobs.  Knowing why is a higher level of awareness than merely doing.

    Sincerely,

    Gene Wirchenko



    In my story, it didn't happen in a meeting (where people are often in a rush and forget things), but an exchange of emails that went through several days (with a lot of CC recipients). But at least they were lucky, because I'm a stubborn person. When I changed the program, I also created that part where exceptions can be configured. They needed that part _soon_.
    </exception></some>
  • Worf (unregistered) in reply to jsmith
    jsmith:

    Reminds me of the source code for the 3d pipes screen saver.  It was in the Windows SDK for a long time.  If you set the elbow style to mixed, occasionally one of the elbows is a teapot.  Well, the source code defines BLUEMOON equal to some three digit number and then at every elbow tests a random number 0-999 to see if it is equal to BLUEMOON.  The source code comments even joke "Once in a blue moon".  I wish I could find it.


    Ah yes, the OpenGL Pipes screensaver. IIRC, one of the command line options or hacks would actually make it happen a bit more frequently.

    Of course, those who have looked at the OpenGL would've probably used GLUT, and there's glutSolidTeapot() and glueWireTeapot calls to generate um... a teapot. Which is probably how it got there in the screensaver...
  • (cs)
    Alex Papadimoulis:

    Stuart's team poured through the code and identified a number of places where this could be happening, patched them all up, and deployed the changes to production. But still, there were more deadlocks.

    After a few more rounds of unsuccessful bug hunting... 

    <font color="#000099">IF</font> (<font color="#000099">CAST</font><font color="#000099">RAND</font>() * 100 + 1 <font color="#000099">AS INT</font>) = 7)
    <font color="#000099">BEGIN
      RAISERROR</font>( <font color="#990000">'Transaction (Process ID %d) was deadlocked on %.*ls resources '</font> +
                 <font color="#990000">'with another process and has been chosen as the deadlock victim. '</font> +
                 <font color="#990000">'Rerun the transaction.'</font>, 13, 10, @@SPID, 8, <font color="#990000">'lock'</font> )
      <font color="#000099">ROLLBACK TRANSACTION
    	RETURN
    END</font>


    WTF?  Didn't Stuart's team of crack bug hunting developers notice the transaction process id was always 13? Seems like that would have been a clue...
  • (cs) in reply to headhigh
    headhigh:
    Alex Papadimoulis:

    Stuart's team poured through the code and identified a number of places where this could be happening, patched them all up, and deployed the changes to production. But still, there were more deadlocks.

    After a few more rounds of unsuccessful bug hunting... 

    <font color="#000099">IF</font> (<font color="#000099">CAST</font><font color="#000099">RAND</font>() * 100 + 1 <font color="#000099">AS INT</font>) = 7)
    <font color="#000099">BEGIN
      RAISERROR</font>( <font color="#990000">'Transaction (Process ID %d) was deadlocked on %.*ls resources '</font> +
                 <font color="#990000">'with another process and has been chosen as the deadlock victim. '</font> +
                 <font color="#990000">'Rerun the transaction.'</font>, 13, 10, @@SPID, 8, <font color="#990000">'lock'</font> )
      <font color="#000099">ROLLBACK TRANSACTION
    	RETURN
    END</font>


    WTF?  Didn't Stuart's team of crack bug hunting developers notice the transaction process id was always 13? Seems like that would have been a clue...


    Wrong. 13=severity, 10=state.
     Original MSDN page:

    RAISERROR ( { msg_id | msg_str | @local_variable }
        { ,severity ,state }
        [ ,argument [ ,...n ] ] )
        [ WITH option [ ,...n ] ]



  • wang (unregistered) in reply to ammoQ

    The real wtf here is that no one spotted that the deadlock report error number was 50,000.  Kind of a giveaway tbh.  Oh, and the +'s would break it - so I doubt that it actually looked like that...

     

     

  • (cs) in reply to wang
    Anonymous:

    The real wtf here is that no one spotted that the deadlock report error number was 50,000.  Kind of a giveaway tbh.  Oh, and the +'s would break it - so I doubt that it actually looked like that...

     

     


    I really think CASTRAND is a better giveaway for the fact that it wasn't EXACTLY written like that ;)

    But why would the +'s break it?
  • wang (unregistered) in reply to impslayer

    Assuming its sql server... it can't concatenate in the raiserror.

    Instead it throws

    Server: Msg 170, Level 15, State 1, Line 1
    Line 1: Incorrect syntax near '+'.

    Take out the string concatenation and you get a nice error.  But as its a user thrown error, with no error number specified it comes out as error 50000. 

    What would have been neater is to call the correct error
    raiserror(1205, 13, 10, @@spid)

    But you cant raiserror under 13000 ... so he had to write his own.

     

     

     

  • (cs) in reply to wang
    Anonymous:

    Assuming its sql server... it can't concatenate in the raiserror.


    WTF? Had no idea RAISERROR choked on string concatenation...
  • (cs)

    I'm sorry to say that I once perpetrated code like this. I had to hook database commit and rollback-before-termination operations across an entire (very large) application. So I hacked the toolchain to transform commits and rollbacks into calls to the appropriate routine, and then, well, the routines themselves, ahem...

    ... so I screwed up a one-line routine, had rollback() do a COMMIT, and then failed to notice for six months. Oddly none of the huge banks on whose sites this thing was running noticed either...

  • (cs) in reply to Nix
    Nix:
    I'm sorry to say that I once perpetrated code like this. I had to hook database commit and rollback-before-termination operations across an entire (very large) application. So I hacked the toolchain to transform commits and rollbacks into calls to the appropriate routine, and then, well, the routines themselves, *ahem*... ... so I screwed up a one-line routine, had rollback() do a COMMIT, and then failed to notice for six months. Oddly none of the huge banks on whose sites this thing was running noticed either...


    They didn't need a rollback in all that time, perhaps? They must have been exceptionally lucky, but I guess it can happen...
  • (cs) in reply to Worf
    Worf:
    jsmith:

    Reminds me of the source code for the 3d pipes screen saver.  It was in the Windows SDK for a long time.  If you set the elbow style to mixed, occasionally one of the elbows is a teapot.  Well, the source code defines BLUEMOON equal to some three digit number and then at every elbow tests a random number 0-999 to see if it is equal to BLUEMOON.  The source code comments even joke "Once in a blue moon".  I wish I could find it.


    Ah yes, the OpenGL Pipes screensaver. IIRC, one of the command line options or hacks would actually make it happen a bit more frequently.

    Of course, those who have looked at the OpenGL would've probably used GLUT, and there's glutSolidTeapot() and glueWireTeapot calls to generate um... a teapot. Which is probably how it got there in the screensaver...


    Ah, yes, the infamous Teapot... As Steve Baker puts it, it's not just any teapot, it's "The" Teapot. It is steeped in mystery and legend, and little is known about it's past. But it lives on, despite the great mysteries, as a standard rendering model in 3D computer graphics. To find out how it came to infiltrate the OpenGL standard API, and how it came to be a long running in-joke in the computer graphics community, we will journey into the Twilight Zone....

Leave a comment on “The Phantom Deadlock”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article