Comment On The Phantom Deadlock

Anyone can be first if they don't take the time to think of anything logical to write.

Promoted so they can't do any more damage... good choice

well it's always good to add testing code in prod, especially when you haven't added it to dev and staging as well [:|]

IF (CASTRAND() * 100 + 1 AS INT) = 7)

missing a '(' aren't you?

Alex Papadimoulis:

... turns out that a DBA had inserted that code on the production (but not in the development or QA) database for "testing" purposes and forgotten to remove it. But, fortunately, that particular DBA was promoted to a lead position, and wouldn't be "down in the code" as much.

So some Paulas get promoted.

"lead"?  As in heavy stuff that makes you sink in water?  No such luck?  Drat!

Sincerely,

Gene Wirchenko

So let's randomly roll back transactions?  IN PRODUCTION???  Talk about Test First(tm).

editing live code is so much fun!!!  i once ssh'ed into a production webserver, modifieid some code hastily, escaped :wq and exited.  i reloaded the page and, wham!  website destroyed!!!!!  man i got email ever so quickly....and funny thing is, the site wasnt even used that much.....typical  

Not 6, 7!! 7 chipmunks twirlin' on a branch, eatin' lots of sunflowers on my uncle's ranch.
You know that old children's tale from the sea.

Step into my office!

Gene Wirchenko:

Alex Papadimoulis:

... turns out that a DBA had inserted that code on the production (but not in the development or QA) database for "testing" purposes and forgotten to remove it. But, fortunately, that particular DBA was promoted to a lead position, and wouldn't be "down in the code" as much.

So some Paulas get promoted.

"lead"?  As in heavy stuff that makes you sink in water?  No such luck?  Drat!

Sincerely,

Gene Wirchenko

"Lead" as in the stuff in the paint chips that Paulas seem to have consumed as children.

Manni:

Not 6, 7!! 7 chipmunks twirlin' on a branch, eatin' lots of sunflowers on my uncle's ranch. You know that old children's tale from the sea.

Step into my office!

Seven is a number with great significance:

There are 7 days in a week,

The lunar cycle is divisible by 7 (28 days),

there were 7  Dwarfs,

et cetera, et cetera

So let's see if I got this right, every one out of ~100 transactions are randomly rolled back? What purpose could that possibly serve even in a testing environment? The only thing I can think of is to test an application's error handling capabilities by randomly sending it an error. But still, why use a deadlock exception? How about "RAISEERROR( 'This transaction has been randomly nuked ... please try again' ..." instead.

This company apparently uses the opposite of the "cream rises to the top" method for promoting people from within.  They instead believe that shit floats, and therefore should be moved up.

I'd be more interested in what the DBA was trying to test... was he unconvinced that ROLLBACK TRANSACTION was implemented properly by the vendor?!?  And, what, he wanted to be sure it would work if executed at random intervals only?!?

Nah, this is kind of a weak WTF... stupid test code gets left in prod code all the time no matter where you work.  It shouldn't happen, but it does.  The WTF is what the code was meant for in the first place??

WTF does not do this justice.  We need an acronym with a lot more cursing.

Promoted!

Ah, The Peter Principle at its finest.

I used to enjoy reading the regular column Dr. Laurence J.Peter wrote for Psychology Today magazine.

Alex Papadimoulis:

IF (CASTRAND() * 100 + 1 AS INT) = 7)

BEGIN

  RAISERROR( 'Transaction (Process ID %d) was deadlocked on %.*ls resources ' +

             'with another process and has been chosen as the deadlock victim. ' +

             'Rerun the transaction.', 13, 10, @@SPID, 8, 'lock' )

  ROLLBACK TRANSACTION

	RETURN

END

The easy fix is, of course, to change that to CASTRAND() * 10000.  That way you will only have to roll back 1 in every 10,000 transactions, which is a much more manageable number.

This is a weird one.  I cannot fathom what they are testing by manually
raising errors masquerading as deadlocks.  Are they trying to be sure
that the database and application code is written to gracefully handle
all possible deadlocks and/or rolled back transactions?  Why not
actually trap *real* deadlocks and consider actually troubleshooting
real ones and writing code that *avoids* them?  



I just can't comprehend the thought process behind this one, even more than the usual WTF's we see here from day to day.

Jeff S:

This is a weird one.  I cannot fathom what they are testing by manually
raising errors masquerading as deadlocks.  Are they trying to be sure
that the database and application code is written to gracefully handle
all possible deadlocks and/or rolled back transactions?  Why not
actually trap *real* deadlocks and consider actually troubleshooting
real ones and writing code that *avoids* them?  



I just can't comprehend the thought process behind this one, even more than the usual WTF's we see here from day to day.

In a well written application, it's pretty difficult to deliberately
cause a real deadlock. But it can happen. For that reason, it can make
some sense to generate artificial error messages like given in the WTF
to check that the deadlock handling code is working correctly. That
said, I would rather insert code that always throws the error, test the error handling code, and remove the artificial error creation code after that.

ParkinT:

Promoted!

Ah, The Peter Principle at its finest.

I used to enjoy reading the regular column Dr. Laurence J.Peter wrote for Psychology Today magazine.

Let's hope that it's an example of the reverse Peter Principle
where the incompetent are promoted to a level of competence.

Alex Papadimoulis:

Stuart's team poured through the code

I can see it now... coders digitised in some kind of transporter,
turned into liquid binary and poured through the code... of course, I'd
even argue that you can't pore "through" something, but you can pore
over something.



Funny story.. Cisco once sent me a proposal - written by some young pup
of a pre-sales engineer, who talked about "...on a 45MB back-hall..."



laughed my pedantic socks off so I did...

The +1 is completely unneccesary, even if Rand gives you a number starting at zero.

Wait!  You mean DBA's can get promoted?  And this is how to do it??  Man, I am gonna write me into the corner office this afternoon......

versatilia:

Alex Papadimoulis:

Stuart's team poured through the code

I can see it now... coders digitised in some kind of transporter, turned into liquid binary and poured through the code... of course, I'd even argue that you can't pore "through" something, but you can pore over something.

Funny story.. Cisco once sent me a proposal - written by some young pup of a pre-sales engineer, who talked about "...on a 45MB back-hall..."

laughed my pedantic socks off so I did...

That is funny!

You have hit on a "sore point" of mine.  There are many criminal abuses asserted on our language (well, I guess it is YOUR language - being from the UK) and I find myself cringing often when they surface.

I have mentioned before on this forum how comical I find it to read software developer positions advertised in the local paper that require:

"experience in Pearl and Sequel"

Developing your own error logging/notification system is a WTF. "not invented here" / Yet Another Error Logging System. Although that DBA should be fired for trying stupid stuff like that on a PRODUCTION machine.

marvin_rabbit:

Alex Papadimoulis:

IF (CASTRAND() * 100 + 1 AS INT) = 7)

BEGIN

  RAISERROR( 'Transaction (Process ID %d) was deadlocked on %.*ls resources ' +

             'with another process and has been chosen as the deadlock victim. ' +

             'Rerun the transaction.', 13, 10, @@SPID, 8, 'lock' )

  ROLLBACK TRANSACTION

	RETURN

END

The easy fix is, of course, to change that to CASTRAND() * 10000.  That way you will only have to roll back 1 in every 10,000 transactions, which is a much more manageable number.

LOL...

seriously, I can't imagine why you would put such message and roll back the transaction randomly

reminds me of some co-workers who struggled for days trying to find a NULL in some field, the queries just wouldn't work as expected... until someone decided to try searching for 'NULL'

This looks to me like a true -3 on the Capability Im-Maturity Model http://www.stsc.hill.af.mil/crosstalk/1996/11/xt96d11h.asp (Undermining / Sabatoge).  No matter how hard I try to find a valid reason for this I cannot make sense out of it.

Oh, cmon, like none of you have ever edited anything in production? This is a nasty one but get off your high horse will you!

You think that might have something to do with the person on the other end of the phone writing down exactly what they hear?

Anonymous:

Oh, cmon, like none of you have ever edited anything in production? This is a nasty one but get off your high horse will you!

Not like that, no.

I have had to edit data, because of a failure, but changing program behaviour on a production system without testing first?  No.

Sincerely,

Gene Wirchenko

Anonymous:

This looks to me like a true -3 on the Capability Im-Maturity Model http://www.stsc.hill.af.mil/crosstalk/1996/11/xt96d11h.asp (Undermining / Sabatoge).  No matter how hard I try to find a valid reason for this I cannot make sense out of it.

A great read!  Thank you for the pointer.

Sincerely,

Gene Wirchenko

Anonymous:

Oh, cmon, like none of you have ever edited anything in production? This is a nasty one but get off your high horse will you!

I once hex-editted a binary in production in order to fix an Sql query that was failing.  Thank goodness the fix was to delete chars (well, overwrite with spaces), instead of adding them...

Gene Wirchenko:

Anonymous:

Oh, cmon, like none of you have ever edited anything in production? This is a nasty one but get off your high horse will you!

Not like that, no.

I have had to edit data, because of a failure, but changing program behaviour on a production system without testing first?  No.

Sincerely,

Gene Wirchenko

PFFFFTTT!!!...

newbies!!

Anonymous:

Anonymous:

Oh, cmon, like none of you have ever edited anything in production? This is a nasty one but get off your high horse will you!

I once hex-editted a binary in production in order to fix an Sql query that was failing.  Thank goodness the fix was to delete chars (well, overwrite with spaces), instead of adding them...

been there, done that[8-|]

Now, surely you haven't heard about something as ugly as doing a blooper on a Production database ... in MySQL, using MyISAM tables. Oops ... no ROLLBACK!

Non-Transactional Databases are the worst WTF ever concieved, except maybe for XML-powered databases (I don't mean xml-supporting DBMS, I mean DBMS that actually STORE the data in XML).

And yes, I know there is InnoDB for MySQL, but still, they didn't just stick in Transactional stuff in the beginning, they actually promoted non-transactional production systems when they started up!!! However, it seems they have vaporized such remarks without a trace from their documentation.

limelight:

So let's see if I got this right, every one out of ~100 transactions are randomly rolled back? What purpose could that possibly serve even in a testing environment? The only thing I can think of is to test an application's error handling capabilities by randomly sending it an error. But still, why use a deadlock exception? How about "RAISEERROR( 'This transaction has been randomly nuked ... please try again' ..." instead.

Yea, that's the WTF. With that wording, how would you be able to differentiate the actual errors from the one's you're generating by yourself? Otherwise, I'm sure you could come up with reasons that you'd need to test an application and make sure it can do a bunch of rollbacks while under a decent load...You should always make sure your test code isn't left lying around though. That many errors should never have gotten through any sort of quality process.

Satanicpuppy:

limelight:

So let's see if I
got this right, every one out of ~100 transactions are randomly rolled
back? What purpose could that possibly serve even in a testing
environment? The only thing I can think of is to test an application's
error handling capabilities by randomly sending it an error. But still,
why use a deadlock exception? How about "RAISEERROR( 'This transaction
has been randomly nuked ... please try again' ..." instead.

Yea,
that's the WTF. With that wording, how would you be able to
differentiate the actual errors from the one's you're generating by
yourself? Otherwise, I'm sure you could come up with reasons that you'd
need to test an application and make sure it can do a bunch of
rollbacks while under a decent load...You should always make sure your
test code isn't left lying around though. That many errors should never
have gotten through any sort of quality process.

Deadlocks are probably handled differently than all other errors. It's
quite reasonable to simply retry a transaction that failed because of a
deadlock, probably even without reporting the problem to the end-user.

Satanicpuppy:

limelight:

So let's see if I got this right, every one out of ~100 transactions are randomly rolled back? What purpose could that possibly serve even in a testing environment? The only thing I can think of is to test an application's error handling capabilities by randomly sending it an error. But still, why use a deadlock exception? How about "RAISEERROR( 'This transaction has been randomly nuked ... please try again' ..." instead.

Yea, that's the WTF. With that wording, how would you be able to differentiate the actual errors from the one's you're generating by yourself? Otherwise, I'm sure you could come up with reasons that you'd need to test an application and make sure it can do a bunch of rollbacks while under a decent load...You should always make sure your test code isn't left lying around though. That many errors should never have gotten through any sort of quality process.

Maybe they had to do it that way because some code somewhere parses the error text to figure out what has gone wrong - wouldn't surprise me at all, not around here anyway...



(Or maybe it's just sabotage/criminal stupidity.)

BlackTigerX:

Gene Wirchenko:

Anonymous:

Oh, cmon, like none of you have ever edited anything in production? This is a nasty one but get off your high horse will you!

Not like that, no.

I have had to edit data, because of a failure, but changing program behaviour on a production system without testing first?  No.

Sincerely,

Gene Wirchenko

PFFFFTTT!!!...

newbies!!

Yea, I've had to shoot from the hip a few times myself as well. That's probably about the most exciting (though not necessarily exciting in a good way) part of a programmers job. The business I'm in right now is SO time and schedule driven that stuff has to be flipped on the fly WAY more often than I would like. Makes your heart beat fast.

You can't tell me that nobody grepped the code for that error message to find out where it was coming from. And when it didn't appear in dev or test, grep the production code as well. WTF?

Maybe that was their standard 'deadlock' error message, and could have come from several places in the code. In that case, why didn't their 'standard' deadlock error message report the file and line in the code that it came from? WTF?

Sounds like the development and debugging practices of the team as a whole are a bit special.

It looks like he injected the random error so that he could take credit for removing the "problem" later on.

I've worked for a couple of -1 organizations (as we have all), which I think is the norm for most of the small IT shops.  I can't imagine how bad a -3 organization is (indicitive of a medium-to-large organization that has a long lifetime).

You hear the words "burn out" often and you also see lots of people quit (enough that there's a job posting for an IT position up on the monsterboard to replace someone who left at least 1x a quarter) to go to greener pastures in a -1.

I wonder how this guy managed to stay at a -3 organization?

Anonymous:

This looks to me like a true -3 on the Capability Im-Maturity Model http://www.stsc.hill.af.mil/crosstalk/1996/11/xt96d11h.asp (Undermining / Sabatoge).  No matter how hard I try to find a valid reason for this I cannot make sense out of it.

Satanicpuppy:

BlackTigerX:

Gene Wirchenko:

Anonymous:

Oh, cmon, like none of
you have ever edited anything in production? This is a nasty one but
get off your high horse will you!

Not like that, no.

I
have had to edit data, because of a failure, but changing program
behaviour on a production system without testing first?  No.

Sincerely,

Gene Wirchenko

PFFFFTTT!!!...

newbies!!

Yea, I've had to shoot from the hip a
few times myself as well. That's probably about the most exciting
(though not necessarily exciting in a good way) part of a programmers
job. The business I'm in right now is SO time and schedule driven that
stuff has to be flipped on the fly WAY more often than I would like.
Makes your heart beat fast.

Sometimes a change is so urgent that there is no other chance than a
shot from the hip. But if that happens too often, it's a sign of a bad
(or non-existing) management. The most likely case of such an urgent
modification is when a change was made that was not good, eighter
because it wasn't proberly tested or because the specification for that
change was wrong. Some week ago, a client demanded "prevent to program
from doing X". I asked "no exceptions from that rule?". They: "No". I:
"Are you sure about that? Think of (common exceptions at other
clients).".

They: "No".



Guess.

Tell me, without going to the system tables in SQL Server or running a cursor to call sp_helptext on all the stored procs, how do you quickly "grep" TSQL stored procs?

That's right.  You can't.  You have to write one or query the underlying schema manually since it's not intuitively easy.

USE [dbname]

SELECT SPECIFIC_NAME, ROUTINE_DEFINITION

FROM INFORMATION_SCHEMA.ROUTINES

WHERE

ROUTINE_DEFINITION LIKE '%foo%'

Anonymous:

You can't tell me that nobody grepped the code for that error message to find out where it was coming from. And when it didn't appear in dev or test, grep the production code as well. WTF?

Maybe that was their standard 'deadlock' error message, and could have come from several places in the code. In that case, why didn't their 'standard' deadlock error message report the file and line in the code that it came from? WTF?

Sounds like the development and debugging practices of the team as a whole are a bit special.

(or you can script out all the stored procs from SQLEM)

BlackTigerX:

Gene Wirchenko:

Anonymous:

Oh, cmon, like none of you have ever edited anything in production? This is a nasty one but get off your high horse will you!

Not like that, no.

I have had to edit data, because of a failure, but changing program behaviour on a production system without testing first?  No.

PFFFFTTT!!!...

newbies!!

Who?  Certainly not me.

Sincerely,

Gene Wirchenko

Select     object_name(id)
from syscomments
where text like '%foo%'

Ken:

Select     object_name(id)
from syscomments
where text like '%foo%'

Stupid forum

Anonymous:

You can't tell me that nobody grepped the code
for that error message to find out where it was coming from. And when
it didn't appear in dev or test, grep the production code as well. WTF?


Maybe that was their standard 'deadlock' error message, and
could have come from several places in the code. In that case, why
didn't their 'standard' deadlock error message report the file and line
in the code that it came from? WTF?

Sounds like the development and debugging practices of the team as a whole are a bit special.

I guess this is the standard error message coming from the database
system. You do not normaly expect that the application fakes that. It's
like grepping for "general protection failure" in the source of a
Windows program.

ammoQ:

Jeff S:

This is a weird one.  I cannot fathom what they are testing by manually
raising errors masquerading as deadlocks.  Are they trying to be sure
that the database and application code is written to gracefully handle
all possible deadlocks and/or rolled back transactions?  Why not
actually trap *real* deadlocks and consider actually troubleshooting
real ones and writing code that *avoids* them?  



I just can't comprehend the thought process behind this one, even more than the usual WTF's we see here from day to day.

In a well written application, it's pretty difficult to deliberately
cause a real deadlock. But it can happen. For that reason, it can make
some sense to generate artificial error messages like given in the WTF
to check that the deadlock handling code is working correctly. That
said, I would rather insert code that always throws the error, test the error handling code, and remove the artificial error creation code after that.

More importantly, if you insert temp code into a production system, then wrap it in "if user = the_consultant then { ... }" so that it doesn't affect the regular users.

limelight:

So let's see if I got this right, every one out
of ~100 transactions are randomly rolled back? What purpose could that
possibly serve even in a testing environment? The only thing I can
think of is to test an application's error handling capabilities by
randomly sending it an error. But still, why use a deadlock exception?
How about "RAISEERROR( 'This transaction has been randomly nuked ...
please try again' ..." instead.

Because that would make sense, so it wouldn't be here, so you wouldn't
have seen it, so you wouldn't have been able to complain about it and
then we'd have a paradox and the universe would explode.