Stuart's team had just finished up developing an error notification system to log and email when certain exceptions were thrown in the application. After deployment, they noticed that one of the applications was having a fair amount of trouble with database deadlocks. Stuart's team poured through the code and identified a number of places where this could be happening, patched them all up, and deployed the changes to production. But still, there were more deadlocks.

After a few more rounds of unsuccessful bug hunting, they were finally given permission to attach a trace to the database server to watch for deadlocks. Not a single deadlock appeared in the trace. The logs, however, were continuing to fill with deadlocks. Stuart nervously checked a data validation trigger and found the problem ... 

IF (CASTRAND() * 100 + 1 AS INT) = 7)
BEGIN
  RAISERROR( 'Transaction (Process ID %d) was deadlocked on %.*ls resources ' +
             'with another process and has been chosen as the deadlock victim. ' +
             'Rerun the transaction.', 13, 10, @@SPID, 8, 'lock' )
  ROLLBACK TRANSACTION
	RETURN
END

... turns out that a DBA had inserted that code on the production (but not in the development or QA) database for "testing" purposes and forgotten to remove it. But, fortunately, that particular DBA was promoted to a lead position, and wouldn't be "down in the code" as much.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!