Comment On Logged to Death

"APPL IS COMPLTLY FRIED," Damien's emergency pager read, "CANT EVEN LOGON TO SRVR!!!" Damien dragged himself out of bed and VPN'ed on in. The middle-of-the-night page did not lie: any attempt at accessing the application resulted in an incomprehensible "Internal Framework Error," and the server prevented any logon attempts with something to the effect of "cannot load security profile." Fortunately, the logs were available ... all sixty-eight gigabytes of them. Damien was able to kill the logs, which were consuming every bit of free space on the drive and every bit of available memory, queued to write to disk. [expand full text]
« PrevPage 1 | Page 2 | Page 3Next »

Re: Logged to Death

2005-12-13 13:41 • by Sweets
huh......

Re: Logged to Death

2005-12-13 13:41 • by Darax The Good

I would have written 16 k like this:


 


string My_16_6k = "kkkkkkkkkkkkkkkk";


Much cleaner.

Re: Logged to Death

2005-12-13 13:42 • by Darax The Good
53227 in reply to 53226
Darax The Good:

I would have written 16 k like this:


 


string My_16_k = "kkkkkkkkkkkkkkkk";


Much cleaner.



 


Oops, Typo...fxd

Re: Logged to Death

2005-12-13 13:42 • by KraGiE
oh don't get me started on logs!!!  Had a guy try to parse log files to repair the data his app broke.  omg.  Horrible times

Re: Logged to Death

2005-12-13 13:42 • by ammoQ
*lol* too much logging is worse than no logging at all ;-)

Re: Logged to Death

2005-12-13 13:42 • by Brendan Kidwell
Uhm... what? I'm confused. What purpose could this serve other than sabotage? Not that I'm accusing anyone of sabotage (always give the benefit of the doubt), but what's going on here?

Re: Logged to Death

2005-12-13 13:46 • by haveworld
Alex Papadimoulis:
to ensure that the log properly wrote out 16k ... every single second


WHY?

Re: Logged to Death

2005-12-13 13:50 • by ammoQ
53233 in reply to 53230
Brendan Kidwell:
Uhm... what? I'm confused. What purpose
could this serve other than sabotage? Not that I'm accusing anyone of
sabotage (always give the benefit of the doubt), but what's going on
here?




To make sure the buffer gets written to the disk. Writter.flush() anyone?

Re: Logged to Death

2005-12-13 13:51 • by Jason
53234 in reply to 53231
How was that not noticed during testing?

Re: Logged to Death

2005-12-13 13:55 • by diaphanein
53235 in reply to 53234
The real purpose of this is to make sure that each log message is easy to spot.  You know...between all the asterisks. [;)]

Re: Logged to Death

2005-12-13 13:57 • by MikeB
Oh, there was sabotage all right. A career has been sabotaged.

Re: Logged to Death

2005-12-13 14:01 • by Dave

Log...it's big, it's heavy, it's wood. It's better than bad, it's good!


(Apologies to Ren and Stimpy.)


 

Re: Logged to Death

2005-12-13 14:04 • by tmountjr
53238 in reply to 53234
Using some round numbers, this was going on non-stop for 2 months, 3 days, 6 hours, 12 minutes, and 55 seconds before it crashed...at least. One of the real WTFs is that this was going on for so long and no one noticed their server storage slowly grinding to a halt.

Re: Logged to Death

2005-12-13 14:05 • by DiamondDave
whatever happened to code reviews?

Re: Logged to Death

2005-12-13 14:06 • by Athas
53240 in reply to 53237
I do not believe this. Not even the dimmest programmers I know would be
able to perform such a feat, without knowing that is was incredibly
stupid? And how come it wasn't discovered earlier? Doesn't anyone just skim the logs at that company?





Re: Logged to Death

2005-12-13 14:06 • by Xepol
This, children, is why you meet "de-hirees" as the come in the door with their last 2 weeks pay and large security guards rather than give them the day (or worse the next 2 weeks) to "clean up" and such.

Imminent unemployement can make people stupid...

Frankly I am amazed that he managed to get as close to the server as he did even with a VPN.

Mind you, this might have concievably been a job security feature... "The system doesn't work right when bob isn't here"... Sure, because no one but bob knows to delete the logs every 3 hours or so...

Re: Logged to Death

2005-12-13 14:12 • by BlackTigerX

ah!... reminds me when the hard drive of a soon-to-be "de-hired" coworker, became completly corrupt the day before he left!!, this co-worker was nice enough that he spend the last day "working", trying to recover his damaged drive


arrrrgghh!!!


and the guy was so worried that any code that he created that wasn't for the company wouldn't be anywhere to be found, like he created the next killer application...


what a loser!

Re: Logged to Death

2005-12-13 14:16 • by kipthegreat
53243 in reply to 53241
Xepol:
This, children, is why you meet "de-hirees" as the come in the door with their last 2 weeks pay and large security guards rather than give them the day (or worse the next 2 weeks) to "clean up" and such.

Imminent unemployement can make people stupid...

Frankly I am amazed that he managed to get as close to the server as he did even with a VPN.

Mind you, this might have concievably been a job security feature... "The system doesn't work right when bob isn't here"... Sure, because no one but bob knows to delete the logs every 3 hours or so...


At the two companies I've worked for, VPN access could get you to a Remote Desktop or VNC seesion on your PC at the office, where you could then do anything you could normally do while sitting in front of it.

As for your "no one but Bob knows how to delete the logs" comment--  It is probably that several developers trade-off "pager duty".. Steve
has it this week, Dave next week, etc... Not that "bob" was the only one who
was capable of deleting the log files.  He was just the one on call that night.

Re: Logged to Death

2005-12-13 14:22 • by kipthegreat
Alex Papadimoulis:

A bit of investigation the next day revealed the culprit: the LogFillerPlugin class. Written by a "de-hired" coworker, its purpose was surprisingly not sabotage, but to ensure that the log properly wrote out 16k ... every single second ...



Forgive me for leaving my sarcasm detector at home today.  Is Alex being sarcastic or was this really not sabotage?  I don't even see a bad reason for this being there if not for sabotage..

Also, when he says this was written by a "de-hired" employee, does that mean the guy was told he had two more weeks or whatever, and in that time period he wrote this code?  In that case, this would almost certainly be sabotage.  Or does it mean that the guy wrote this code with no knowledge that he would soon be fired?  Or, third possibility, he was fired, but before passwords were reset he VPN'ed in and did this?

There's a lot of ambiguity here...

Re: Logged to Death

2005-12-13 14:44 • by GoatCheez
53245 in reply to 53240
This guy has to be completely retarded. First off, the whole thing is just stupid. Then, somehow, he knows that creating the string like the way he does is faster than making a for loop to create it. But THEN, even though it's all FILLER, he puts NEWLINES into it at the END OF EVERY LINE.... www.WTF.com.

Re: Logged to Death

2005-12-13 14:44 • by Richard Nixon
53246 in reply to 53243
kipthegreat:

As for your "no one but Bob knows how to delete the logs" comment--  It is probably that several developers trade-off "pager duty".. Steve
has it this week, Dave next week, etc... Not that "bob" was the only one who
was capable of deleting the log files.  He was just the one on call that night.


I believe you completely misunderstood the point that was being made. Try rereading it a few times.

Sincerely,
Richard Nixon

Re: Logged to Death

2005-12-13 14:47 • by Wolf550e
53247 in reply to 53230
Brendan Kidwell:
Uhm... what? I'm confused. What purpose could this serve other than sabotage? Not that I'm accusing anyone of sabotage (always give the benefit of the doubt), but what's going on here?

Maybe the saboteur he didn't know how to flush to disk (fsync) and thought if he filled a whole file system block it would get written to disk? But why not generate the filler with a loop? Personally, I would've generated pseudo-random filler on demand, to circumvent compression on the storage level. </kidding>

signed,
An IT code monkey who has seen too many idiots armed with VM oriented languages.

Re: Logged to Death

2005-12-13 14:54 • by Volmarias
53248 in reply to 53246
He knows enough to make it final, static, and private, but he doesn't know how to flush?

Actually, that sounds like a few people I lived with in college... +o(

Re: Logged to Death

2005-12-13 14:55 • by Volmarias
53249 in reply to 53248
and by +o( I mean [+o(]

I miss you, preview!

Re: Logged to Death

2005-12-13 14:56 • by Volmarias
53250 in reply to 53249
Oh, you're fucking shitting me. It replaces the goddamned emoticon with HTML that DOESN'T EVEN WORK BECAUSE THE SERVER ESCAPES IT

Re: Logged to Death

2005-12-13 15:00 • by skicow
53251 in reply to 53236

MikeB:
Oh, there was sabotage all right. A career has been sabotaged.


"You say sabotage, I say sabatage" - William Shatner.

Re: Logged to Death

2005-12-13 15:06 • by TankerJoe
53253 in reply to 53250
Volmarias:
Oh, you're fucking shitting me. It replaces the
goddamned emoticon with HTML that DOESN'T EVEN WORK BECAUSE THE SERVER
ESCAPES IT




That's because god hates people who use emoticons.

Re: Logged to Death

2005-12-13 15:12 • by WTFer
53254 in reply to 53250
Volmarias:
Oh, you're fucking shitting me. It replaces the
goddamned emoticon with HTML that DOESN'T EVEN WORK BECAUSE THE SERVER
ESCAPES IT




Welcome to the Daily WTF.

Re: Logged to Death

2005-12-13 15:14 • by IceBox13
53255 in reply to 53248
Volmarias:
He knows enough to make it final, static, and private, but he doesn't know how to flush? Actually, that sounds like a few people I lived with in college... +o(

Re: Logged to Death

2005-12-13 15:15 • by IceBox13
53256 in reply to 53248

Volmarias:
He knows enough to make it final, static, and private, but he doesn't know how to flush? Actually, that sounds like a few people I lived with in college... +o(


If your college buddies didn't know how to flush...it wasn't really private then either, was it?

Re: Logged to Death

2005-12-13 15:27 • by Xargon
53257 in reply to 53234
Anonymous:
How was that not noticed during testing?

Probably because testing consisted of a run over a short timescale, and no one cared if the log was on the order of a megabyte during testing.

I laughed long and hard at this one.

Re: Logged to Death

2005-12-13 15:29 • by Satanicpuppy
Yea, I've been beating my brain trying to come up with any non-malicious reason for putting a zillion "*"'s in a log file, and the only other thing I can come up with is utter stupidity. If nothing else, it'd make your logs utterly unreadable, even using grep because the nearest meaningful lines would be 100 lines of *'s away.

This guy should be taken out and shot.

Re: Logged to Death

2005-12-13 15:32 • by cconroy
53260 in reply to 53256
IceBox13:

Volmarias:
He knows enough to make it final, static, and private,
but he doesn't know how to flush? Actually, that sounds like a few
people I lived with in college... +o(


If your college buddies didn't know how to flush...it wasn't really private then either, was it?





It would certainly make it static, though...



Re: Logged to Death

2005-12-13 15:33 • by HAK
And here is a prime example of why I follow TDWTF .... to pop my
delusional bubbles of what really goes on in the real world, outside of
university ......

Re: Logged to Death

2005-12-13 15:34 • by FrostCat
53262 in reply to 53242
BlackTigerX:

ah!... reminds me when the hard drive of a
soon-to-be "de-hired" coworker, became completly corrupt the day
before he left!!, this co-worker was nice enough that he spend the last
day "working", trying to recover his damaged drive


arrrrgghh!!!


and the guy was so worried that any code that he created that wasn't
for the company wouldn't be anywhere to be found, like he created the
next killer application...


what a loser!





We once had a developer add an ACL to his XP box so that Everyone had
read-only access to the hard drive, and only local administrators has
write access.  That's a fun one to get out of.  knoppix-std
wouldn't work because the machine only had SATA drives.

Re: Logged to Death

2005-12-13 15:35 • by kipthegreat
53263 in reply to 53246
Richard Nixon:
kipthegreat:

As for your "no one but Bob knows how to delete the logs" comment--  It is probably that several developers trade-off "pager duty".. Steve
has it this week, Dave next week, etc... Not that "bob" was the only one who
was capable of deleting the log files.  He was just the one on call that night.


I believe you completely misunderstood the point that was being made. Try rereading it a few times.

Sincerely,
Richard Nixon


My bad Mr. President.

Re: Logged to Death

2005-12-13 15:37 • by Hugo
53264 in reply to 53244

kipthegreat:
Also, when he says this was written by a "de-hired" employee, does that mean the guy was told he had two more weeks or whatever, and in that time period he wrote this code?  In that case, this would almost certainly be sabotage.  Or does it mean that the guy wrote this code with no knowledge that he would soon be fired?  Or, third possibility, he was fired, but before passwords were reset he VPN'ed in and did this?


You forget the fourth possibility - the guy was fired because he wrote this code.

Re: Logged to Death

2005-12-13 15:38 • by Maurits
53265 in reply to 53261
Why didn't they monitor the server hard drive space?  If a drive hits 80% full, you send an email to the admins...



Oh, and, yes, emoticons make baby Jesus cry :'(

Re: Logged to Death

2005-12-13 15:39 • by WTFer
53266 in reply to 53263
kipthegreat:
Richard Nixon:
kipthegreat:

As
for your "no one but Bob knows how to delete the logs" comment-- 
It is probably that several developers trade-off "pager duty".. Steve
has it this week, Dave next week, etc... Not that "bob" was the only
one who
was capable of deleting the log files.  He was just the one on
call that night.


I believe you completely misunderstood the point that was being made. Try rereading it a few times.

Sincerely,
Richard Nixon


My bad Mr. President.


He is not the real president, you know,  he resigned.

Re: Logged to Death

2005-12-13 15:51 • by IceBox13
53268 in reply to 53264
Hugo:

kipthegreat:
Also, when he says this was written by a "de-hired" employee, does that mean the guy was told he had two more weeks or whatever, and in that time period he wrote this code?  In that case, this would almost certainly be sabotage.  Or does it mean that the guy wrote this code with no knowledge that he would soon be fired?  Or, third possibility, he was fired, but before passwords were reset he VPN'ed in and did this?


You forget the fourth possibility - the guy was fired because he wrote this code.



It's a possibility.  Alex's post doesn't give any indication as to the timeframe of the de-hiring.  This would definitely warrant it.

Re: Logged to Death

2005-12-13 16:00 • by Coughptcha
53269 in reply to 53238
tmountjr:
Using some round numbers, this was going on
non-stop for 2 months, 3 days, 6 hours, 12 minutes, and 55 seconds
before it crashed...at least. One of the real WTFs is that this was
going on for so long and no one noticed their server storage slowly
grinding to a halt.
I'm not sure about the math you used.



I get 68Million K / 16K = 2.5Million seconds

2.5M s = 70833.33 minutes = 1180.55 hours = 49.1898 days



Or about 2% of the available disk space every day.  Still a pretty
slow rate of fill.  And a pretty poor set of testing if you don't
know how quickly your resources are going to be depleted.

Re: Logged to Death

2005-12-13 16:00 • by kipthegreat
53270 in reply to 53266
WTFer:
kipthegreat:
Richard Nixon:
kipthegreat:

As
for your "no one but Bob knows how to delete the logs" comment-- 
It is probably that several developers trade-off "pager duty".. Steve
has it this week, Dave next week, etc... Not that "bob" was the only
one who
was capable of deleting the log files.  He was just the one on
call that night.


I believe you completely misunderstood the point that was being made. Try rereading it a few times.

Sincerely,
Richard Nixon


My bad Mr. President.


He is not the real president, you know,  he resigned.


"After a president of the U.S. leaves office, the title "President" continues to be applied to that person the rest of his life."
  -- http://en.wikipedia.org/wiki/President_of_the_United_States

Although, Dick is no longer alive so.. whatever..

Re: Logged to Death

2005-12-13 16:07 • by Anonymous
The real WTF there is that the type modifiers are backwards. It should be "private static final" not "final static private".

Re: Logged to Death

2005-12-13 16:12 • by Richard Nixon
53272 in reply to 53263
kipthegreat:
Richard Nixon:
kipthegreat:

As for your "no one but Bob knows how to delete the logs" comment--  It is probably that several developers trade-off "pager duty".. Steve
has it this week, Dave next week, etc... Not that "bob" was the only one who
was capable of deleting the log files.  He was just the one on call that night.


I believe you completely misunderstood the point that was being made. Try rereading it a few times.

Sincerely,
Richard Nixon


My bad Mr. President.


Do you see your error in thought now? I think this way of educating you is the most effective.

Sincerely,
Richard Nixon

Re: Logged to Death

2005-12-13 16:30 • by Kooch
53274 in reply to 53271
Anonymous:
The real WTF there is that the type modifiers
are backwards. It should be "private static final" not "final static
private".




The Java compiler doesn't think they are backwards. Why is that a WTF? Who's definition of forwards and backwards are you using?

Re: Logged to Death

2005-12-13 16:31 • by Andir
Alex Papadimoulis:

"APPL IS COMPLTLY FRIED," Damien's emergency pager read, "CANT EVEN LOGON TO SRVR!!!" Damien dragged himself out of bed and VPN'ed on in.


I don't mean to point out gramatical errors, but "dragged himself out of bed"?  Was it just me or did anyone else have a problem reading this first set of sentences with the combination of poor quoting and then the made word?

Re: Logged to Death

2005-12-13 16:44 • by anon1
53276 in reply to 53275
Anonymous:
Alex Papadimoulis:

"APPL IS COMPLTLY FRIED," Damien's emergency pager read, "CANT EVEN LOGON TO SRVR!!!" Damien dragged himself out of bed and VPN'ed on in.


I don't mean to point out gramatical errors, but "dragged himself out of bed"?  Was it just me or did anyone else have a problem reading this first set of sentences with the combination of poor quoting and then the made word?



Actually "dragged" is correct.


http://dictionary.reference.com/search?q=dragged


 

Re: Logged to Death

2005-12-13 16:50 • by kipthegreat
53277 in reply to 53275
Anonymous:
Alex Papadimoulis:

"APPL IS COMPLTLY FRIED," Damien's emergency pager read, "CANT EVEN LOGON TO SRVR!!!" Damien dragged himself out of bed and VPN'ed on in.


I don't mean to point out gramatical errors, but "dragged himself out of bed"?  Was it just me or did anyone else have a problem reading this first set of sentences with the combination of poor quoting and then the made word?



I'm going to go out on a limb and suggest that perhaps English is not your native language?

Re: Logged to Death

2005-12-13 16:52 • by Pointy Haired Manager
53278 in reply to 53239

DiamondDave:
whatever happened to code reviews?


Yes, I know that solid engineering practices like code reviews actually shorten the development cycle but we don't have time for them, dammit!!!


I am fully aware that quality is free but we can't afford to implement that now, dammit!!!


We are all aware that written requirements that are agreed to and signed off act to keep a project on time and under budget but, dammit, we're already behind schedule and we're bleeding money.


Dammit, man!  Why can't you come up with a suggestion that would actually help us?  Dammit!!

Re: Logged to Death

2005-12-13 17:16 • by DiamondDave
53280 in reply to 53278
Hey, I thought they "de-hired" you!!!
« PrevPage 1 | Page 2 | Page 3Next »

Add Comment