• Derp (unregistered)

    Frist

  • Foo AKA Fooo (unregistered)

    "Steven never could figure out what was causing the file corruption" Please tell me you're joking!

    The moment I read, "Thomas had it set up to run every minute from Crontab", I thought, well every minute seems quite ambitious; wonder if the job will always finish in time.

    I hope you can figure out the rest now.

  • (nodebb)

    " added a new <download*gt; element "

    Ò_o

    No wonder it was breaking...

  • blakeyrat (unregistered)

    A version that actually does some error checks (and doesn't use perlisms from the 90s, such as global filehandles):

    # Name the file after today's date
my $file = "$log_dir/$date.log";
# >> opens the file for append. Linux *always* gets this right.
open my $fh, ">>",  $file or die "Can't open $file: $!";
# Look Ma! No XML!
print $fh "$date $time $downloaded_filename\n" or die "Can't write $file: $!";
close $fh or die "Can't write $file: $!";
# Delete any files that are more than 3 days old
# (assumes $log_dir doesn't contain spaces)
unlink grep { -M > 3 } glob "$log_dir/*.log";

    (I wonder how this code is going to get mangled. There's no preview.)

  • Mike a Person (unregistered)

    SAX ?

  • dkf (unregistered)

    Java's got support for logging in XML, but it does it "right" so it avoids this problem. It does it by using XML tricks to allow a file to contain partial documents; there is a wrapper in another file that effectively just includes the real log file.

    It works. It is correct XML. It also horrifies all sane developers.

  • Ron Fox (google) in reply to Mike a Person

    SAX and Violence.

  • b11c (unregistered) in reply to Ron Fox

    SAX and violins

  • Disappointed (unregistered)

    Since the down times corresponded to Thomas showing up hungover, I expected the point of the story to be that Thomas was manually doing some critical part of the job. Then, he'd go out and get drunk on the weekend and forget to do that critical thing for a day or two and cause the crashes that way.

    Chekhov's gun. Check it out. https://en.wikipedia.org/wiki/Chekhov%27s_gun

  • JG (unregistered) in reply to dkf

    It works. It is correct XML. It also horrifies all sane developers.

    You actually know a sane developer.

    Ph'nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn

  • Church (unregistered)

    As the replacement solution doesn't use XML, it's not enterprise-y enough and should be discarded. Obviously.

  • Petr (unregistered)

    So, their files kept getting corrupted and - instead of finding and fixing the problem - he choose to silence the problem. His flat files do not have forward error correction codes or transaction journal or anything what would make the file more robust - instead a corruption means a consuming application would continue parsing, but it will parse a garbage.

    He truly is a hacker, and a proud one I see. Thankfully we do not have those in our production team.

  • isthisunique (unregistered)

    It's not really unusual to produce machine readable logs.

    XML is pure filth though. I like JSON but UTF8 can be annoying when you might have binary output sneaking in.

  • Quite (unregistered)

    I've always had trouble with Chekhov's gun. While I understand its necessity in order to prevent littering the story with superfluous detail, it sort of gives the game away to point out the props to the reader -- in particular, they're going to know the gun is going to go off and there will be tears before bedtime as a result. So the difficult thing is to make the reader aware of such devices, but not in such a way as to make it obvious that they are the metaphorical keystone.

  • Ron Fox (google) in reply to Disappointed

    His hung-over-drunkenness was still a necessary plot point in order to have him fired so that Steven had to wade into the filth he left behind. Possibly as well, the original code was written wile Thomas was either drunk, or hung over or, now there's a frightening state, both.

  • Carl Witthoft (google) in reply to Quite

    Quote: "I've always had problems with Chekhov's gun" Endquote Well, duh: you're holding it wrong!

  • dudeNumber4 (unregistered)

    I'm confused. The XML was only there to log that the real log file was downloaded? Sort of a meta-log? How is that even useful? You really only need to delete the old files. If a file isn't there, it wasn't downloaded. WTF?

  • grasshoppa (unregistered) in reply to dkf

    "It works. It is correct XML. It also horrifies all sane developers."

    Well, it's a good thing we don't have any of those around here then, isn't it?

  • Herby (unregistered)

    Chekhov's gun?? Yes, Mr. Sulu.

    Yes, XML is terrible. Somebody should go out and shoot it now.

    I'm listening to George Carlin clips, and if he knew about XML, he would have had a whole bit about it and how terrible it is.

  • Dan Mercer (unregistered) in reply to Quite

    For an amusing take on Checkov's gun and other such theatrical devices, rent "Sleuth" (1972) - Laurence Olivier and Michael Caine. Or "Deathtrap" (1982) Michael Caine and Christopher Reeve.

  • Brian Boorman (google) in reply to dudeNumber4

    or.... it was never generated due to some other problem. The question then is: Do you care?

  • TimothyB (unregistered) in reply to Herby

    It's not George Carlin, and it's not about XML, but he might have appreciated this rant on Javascript.

    Go to page 6, and read the two paragraphs starting with the one in the right-hand column that begins “Of course”.

    https://www.usenix.org/system/files/1403_02-08_mickens.pdf

    Or perhaps read through to the sentence “I know that there are people who like JavaScript, and I hope that these people find the mental health services that they so desperately need.”

  • progger (unregistered) in reply to Quite

    Chekhov's phaser, surely..

  • Tristram (unregistered) in reply to Carl Witthoft

    "Quote: "I've always had problems with Chekhov's gun" Endquote Well, duh: you're holding it wrong!"

    I think it should have been, "It must be the radiation."

  • not_ben_lubar (unregistered)

    Am I the only one who, after reading the title, expected something lojban related?

  • Norman Diamond (unregistered)

    "If you say in the first chapter that there is a rifle hanging on the wall, in the second or third chapter it absolutely must go off."

    OK. But when it goes off the wall, where does it go onto next? Don't leave us hanging.

  • Olivier (unregistered) in reply to Brian Boorman

    Actually, the files that downloaded properly are on the disk or are older than 3 days and have been removed already. So, browse the directory if you want to know what files are there.

    You only need to log when there had been an error, and in this case it'd a human that will read the log file, drop the XML part and concentrate on creating useful contents.

  • Yevhenii (unregistered) in reply to Norman Diamond

    No, you silly! In this context "to go off" simply means "to explode". Of course, it's unclear if we really needed a gun to blow up that wall...

  • Beep boop boop (unregistered) in reply to not_ben_lubar

    Na go'i.

  • Disappointed (unregistered)

    I don't object to the bit about Thomas being a drunk, just that the correlation between hangovers and crashes irritating. If the correlation is of no matter, then don't tell me. If it lead to some incorrent action taken, then that would be fine. But, as is, it is a pointless embellishment that doesn't advance the story - it just makes for more text. Rather like most of the Hanzo stories. embellishment for no gain.

  • Ex-lurker (unregistered)

    All these people are talking about Chekov's gun and while nobody explicitly said that Thomas's hangovers are a Chekov's gun, it feels like that is the general assumption.

    IT IS NOT A CHEKOV'S GUN! (Although it is a plot point as Ron Fox said.)

    A CG is an apparently trivial or insignificant element introduced early on that turns out to be vital in the climax of the story. In this "mystery-like" story it would be a CG if at the end Steven were to find out that those hangovers were the actual cause of the file corruption. As it happened, it was just a RED HERRING.

  • PWolff (unregistered) in reply to grasshoppa

    I never could wrap my head around the concept of that so-called sanity everyone is talking about.

    So what.

  • Guest (unregistered) in reply to Dan Mercer

    I think "The Cabin in the Woods" is another good example http://www.imdb.com/title/tt1259521/

  • (nodebb) in reply to Tristram

    I've always had problems with Chekhov's gun"

    Well, duh: you're holding it wrong!

    I think it should have been, "It must be the radiation."

    No, that's Cherenkov's gun.

  • S. Ivar (unregistered) in reply to urkerab

    Pavel Chekov's gun, not Anton's Chekhov's...

  • smtddr (unregistered) in reply to Foo AKA Fooo

    For those that might not be following.... what I assume happened here, is that after about a week the file got so big that it took longer than a minute to process. Then after 60 seconds crontab starts another one and now you have 2 processes doing I/O on the same file, and when that file is XML.... the chances of a parse error due to a race-condition is pretty high and that's probably what happened.

Leave a comment on “Logjam”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article