• Ben (unregistered)

    Andrew takes on a bet, immediately gets in over his head, then gets Christine to do all the work.

    TRWTF is Andrew hasn't gotten a promotion. He's clearly management material.

  • (cs)

    Did he get his two beers? (and give at least 1 to Christine?)

  • Charles F. (unregistered) in reply to Steve The Cynic
    Steve The Cynic:
    Well, then the WTF is that the platform extension directory is writeable by *anything* except installers.
    It shouldn't even be writable by installers. On my production systems, only administrators can modify the contents of that directory. That's just good practice. So yes, you've identified another TRWTF that a developer could quietly and unilaterally put something in the extensions directory.
    Steve The Cynic:
    However, in the case given, the schmucks were apparently reduced to doing something dangerously close to debug-prints in order to have a clue what was going on.
    This statement isn't remotely accurate. Running checksums on files or turning on classloader logging is not anything resembling putting printlns in your code.
    Steve The Cynic:
    And in the article, we saw a good demonstration of why they should not be there.
    Everyone agrees that the JDBC library should not have been put where it was.
    Steve The Cynic:
    There's a case to be made that perhaps the classes loaded from the platform extension directories should be forced to be members of a particular package hierarchy otherwise they won't be loaded. Notably: if it's NOT part of a platform extension package hierarchy, WTF is it doing there? Why would we *want* to load out-of-hierarchy packages from there?
    The definition of being a platform extension is being in that directory. Restricting the package naming would be stupid and removes the discretion of system architects to decide what can and cannot be installed as an extension. There are all kinds of restrictions that can be put on the code loaded from ext, if you want. For example, you can specify that all JARs in the ext directory be digitally signed by specific signers.

    All of this requires reading a great deal more than one page of documentation, a task that was already too hard for the subjects of the story.

    Steve The Cynic:
    Knowing the way that these things work
    ...is not a phrase that describes you in the context of Java systems.
  • Baboon (unregistered)

    The real WTF is Java devs not thinking to look into lib/ext or the endorsed override mechanism ...

    The other WTF is a Java devs not understanding class loading semantics!

  • (cs) in reply to Steve The Cynic
    Steve The Cynic:
    I think you can also make a case that you might want to be able to activate some sort of output from the JVM that shows the class-loading decisions it made when loading each class (aside from dumb built-in things like strings). That would have shown up the issue immediately, because their classes would have been shown loading from the wrong location.

    For all I know, such a thing exists...

    ... and was already mentioned in a comment before:

    pjt33:
    I don't think that enumerating all the possible locations and searching them is the best option. Use 
    -verbose:class
    and get the VM to tell you where it's finding the class.
  • faoileag (unregistered) in reply to no laughing matter
    no laughing matter:
    From the HTML comments of the article: It was a sysadmin who left the company well before this happened. Why would a sysadmin do such changes without instructions from the developers?
    Perhaps the sysadmin knew about the class loading chain in java and did this as a deliberate prank?

    After all, accidentally copying a file to the wrong directory can happen to anybody; it would be hard to prove intent.

  • Charles F. (unregistered) in reply to Steve The Cynic
    Steve The Cynic:
    You think [Java class loading and C/C++ includes] don't both involve searching for an object down a list of file system resources? (directories, .jar files, etc.)
    I know that Java class loading doesn't necessarily involve searching the file system.

    There's no restriction that any Java class be loaded from the file system. I can load classes -- at runtime -- from email attachments if I want to. Last time I checked, there wasn't a C/C++ directive to "go through my inbox and find this as an email attachment."

    Does this flexibility allow for people to do stupid things? Yes, but remember that systems that prevent you from doing stupid things also prevent you from doing clever things.

  • (cs) in reply to Charles F.
    Charles F.:
    Steve The Cynic:
    Well, then the WTF is that the platform extension directory is writeable by *anything* except installers.
    It shouldn't even be writable by installers. On my production systems, only administrators can modify the contents of that directory. That's just good practice. So yes, you've identified another TRWTF that a developer could quietly and unilaterally put something in the extensions directory.
    Read the HTML comments!

    It wasn't a developer, it was a sysadmin who put the jar in the ext directory (and left the company soon after, so you could consider it as a farewell gift).

  • (cs)

    Seeing as it's almost Halloween, two thoughts:

    1. The name should be Mockingjay, not mockingbird.

    2. Christine is the name of Andrew's car, and no good will come of this collaboration.

  • sexist_sid (unregistered) in reply to accident
    accident:
    Did he get his two beers? (and give at least 1 to Christine?)

    And we wonder why women don't go into development when any mention of a female has to bring out puerile jokes about giving one to Christine. I just don't know any more.

    She might be hot though - any pics?

  • (cs) in reply to Charles F.
    Charles F.:
    I can load classes -- at runtime -- from email attachments if I want to. Last time I checked, there wasn't a C/C++ directive to "go through my inbox and find this as an email attachment."
    Well certainly not a standard one, but why not write an extension to GNU-C++ or Clang to load include files from eMails in the SPAM folder?

    Makes it easier to include Viagra into your application!

    Also modifications to the way DLLs are loaded certainly are possible - it sounds wrong to not load them from the homepage of a Device Lending Library, given the name!

  • (cs) in reply to sexist_sid
    sexist_sid:
    She might be hot though - any pics?
    Christine, as requested!
  • (cs) in reply to sexist_sid
    sexist_sid:
    accident:
    Did he get his two beers? (and give at least 1 to Christine?)

    And we wonder why women don't go into development when any mention of a female has to bring out puerile jokes about giving one to Christine. I just don't know any more.

    She might be hot though - any pics?

    Strike that. Reverse it.

  • Charles F. (unregistered) in reply to no laughing matter
    no laughing matter:
    It wasn't a developer, it was a sysadmin who put the jar in the ext directory
    ...probably at the request of a developer.

    Once again, we learn that regardless of the technology, nothing can protect you from the collaboration of people who are completely clueless.

  • Charles F. (unregistered) in reply to no laughing matter
    no laughing matter:
    Well certainly not a standard one, but why not write an extension to GNU-C++ or Clang to load include files from eMails in the SPAM folder?
    This might actually be a current feature -- it would explain a lot.
    no laughing matter:
    Makes it easier to include Viagra into your application!
    That sounds hard.
  • (cs) in reply to Charles F.
    Charles F.:
    no laughing matter:
    Makes it easier to include Viagra into your application!
    That sounds hard.
    This is your last chance. After this, there is no turning back. You take the blue pill – the compilation ends, you wake up in your bed and believe you have built whatever you wanted to build. You take the red pill – you stay in compilation checks, and I show you how fucked up your code is. Remember, all I'm offering is the truth about your coding abilities – nothing more.
  • (cs) in reply to I R BOFH
    I R BOFH:
    “No, it’s not.” Christine, pale, looked to him out of the corner of her eye. “Who put that jar file there in the first place?”

    A deafening peal of thunder answered her nervous question.

    Thor? Zeus? The Thunder King?

    I don't get it.

    Lord Beric Dondarrion

  • (cs) in reply to Steve The Cynic
    Steve The Cynic:
    ANON:
    Nice story, but actually I see no WTF.
    You don't think that arcane rules about how classes get loaded are sufficiently WTFy for you? Think, man, think! The fact that there can be magical invisible objects that silently override the ones you put there explicitly is a WTF of the highest order.

    Actually this one made me think of an Elegant Syntax Error.

    That's the key. That it's silent. You couldn't compile a .NET project with an ambiguous reference.

  • illum (unregistered) in reply to Charles F.
    Charles F.:
    Steve The Cynic:
    It should, at the very least, be located *after* CLASSPATH in the search order, if for no other reason than to allow me to substitute my own version of a class file for debugging purposes
    1. You do not understand what a platform extension is if you think it's something a user application should be able to override.
    Are you suggesting that Steve The Cynic is "a user application"?
  • Robert (unregistered)

    At least he got his beers.

  • Charles F. (unregistered) in reply to chubertdev
    chubertdev:
    You couldn't compile a .NET project with an ambiguous reference.
    Someone's employer is lucky to have a .NET programmer who doesn't understand the difference between compile-time and run-time!
  • Charles F. (unregistered) in reply to illum
    illum:
    Are you suggesting that Steve The Cynic is "a user application"?
    Anything is possible.
  • (cs) in reply to Charles F.
    Charles F.:
    illum:
    Are you suggesting that Steve The Cynic is "a user application"?
    Anything is possible.
    It can happen. Especially here. See what blakeyrat has become in TDWTF forums...
  • (cs) in reply to no laughing matter
    no laughing matter:
    Todays' story is not a WTF, it's a RTFM!

    Almost the same thing as F is equally descriptive!

  • (cs) in reply to Charles F.
    Charles F.:
    chubertdev:
    You couldn't compile a .NET project with an ambiguous reference.
    Someone's employer is lucky to have a .NET programmer who doesn't understand the difference between compile-time and run-time!

    :facepalm:

  • J (unregistered) in reply to Robert
    Robert:
    At least he got his beers.

    Actually, Stan was able to weasel out of it. The bet was whether Andrew would fix it, but he didn't; Christine did.

  • (cs) in reply to Charles F.
    Charles F.:
    That's just the way it's always been done, and you should never expect it to get better. I've known exactly how Java works since birth.

    is pretty much what I'm getting

  • (cs) in reply to ANON
    ANON:
    Steve The Cynic:
    ANON:
    Nice story, but actually I see no WTF.
    You don't think that arcane rules about how classes get loaded are sufficiently WTFy for you? Think, man, think! The fact that there can be magical invisible objects that silently override the ones you put there explicitly is a WTF of the highest order.

    Actually this one made me think of an Elegant Syntax Error.

    So you'd say the WTF is how Java works? Each platform has its own dependency hell. Java didn't find a perfect solution, but neither did anyone else, so how can you blame it?

    .NET found a good solution with multiple loading contexts, loading into isolated application domains, using assembly version redirects, etc.

    It even has a special and quite verbose logging facility you can query for runtime type and assembly binding errors related to API/ABI type compatibility, duplicate type definitions, etc. and in general it will point you straight and squarely at the culprit.

    Krunt:
    Charles F.:
    Jars can be compiled without debugging information, but this is rarely done anymore as it adds relatively little to the class size.

    How does the inclusion of debug information affect performance of a Jar though (if at all)?

    The reason I ask is that it does have an impact on a .NET assembly, and most of us here do not release binaries to production unless they've been compiled to... well, Release mode.

    Compiling with debug symbols (i.e. generating a *.pdb file) has nothing to do with compiling for Debug or Release modes. (We always compile release mode web applications with debug symbols, because that means we can get a sensible stack trace with line numbers and the works in our error logs...)

  • JAPH (unregistered) in reply to faoileag
    faoileag:
    am I right to assume that... ... I, the application developer, do not have any control over what .jar files might exist in jre/lib/ext/?
    Yes.
    faoileag:
    ... the .jar files in jre/lib/ext/ are loaded prior to my application's .jar files, establishing a method-lookup-up precedence chain?
    Yes.
    faoileag:
    ... it is therefore at least theoretically possible that I pick a name for one of my application's .jar files that also names a .jar file in the jre/lib/ext/ directory?

    This is not the problem. The issue arises when a JAR file within jre/lib/ext contains a class of the same name and package as a jar (or class) file in your CLASSPATH. Same idea, but much rarer. This is like having multiple executable files with the same name in your PATH; if you don't specify which one to use, it grabs the first one it finds (hence the helpfulness of whereis and which).

    faoileag:
    ... and that, to stretch the thought experiment even further, both my and the extension's .jar file could export the same methode (e.g. init())?

    In Java, a class cannot be defined in multiple files. The first file with a class with that name and package is used; any other classes are ignored. Of course, if your application uses multiple ClassLoaders, each class loader could have a different class.

    faoileag:
    ... meaning my init() method would have no chance of ever getting called leaving me going quietly insane trying to find out why my application does not work as intended?
    No, this means that your class would have no chance of ever being loaded, quietly driving you insane trying to figure out why your application doesn't work as intended.
    faoileag:
    If the answer to all these questions is "yes", I'll definitely will have to keep that in mind! :-)
    It's always a good idea to know how external resources are linked in your language of choice.
  • (cs) in reply to Charles F.
    Charles F.:
    ...unless you read the fucking documentation.

    Which is fucking incomplete since it does not mention jre/lib/endorsed, which is mainly used to provide patches to the JRE itself. Jars in there will not only override all user classes, but even classes in the JRE itself (rt.jar), and even if the classes have been dumped into a memory-mappable image (classes.jsa) for "class data sharing", the classes in the image are ignored in case you have a class with same name in jre/lib/endorsed.

    Okay, that directory is not intended to be used by anyone except Oracle/Sun themselves, but if someone does, it may stay unnoticed (since you will probably stop after comparing rt.jar and classes.jsa if you find some difference in some java.* class).

    (Yes, I have seen a POC of a Java "virus" here that overrides a class that is loaded during JVM bootstrap and that has not changed since Java 5, by dropping it into jre/lib/endorsed.)

  • Not A Java Dude (unregistered) in reply to Steve The Cynic

    Methinks the reason for something like this -- and I definitely don't KNOW it is the reason -- would be to prevent unauthorized users from throwing in their on "patched" class lib and reading keystrokes, data, etc. lib/ext is (probably) locked down from access by normal accounts and is counted on by the system as being trusted.

    Again, I have no idea, but something like this comes to mind for me.

    Captcha: odio "On one hand this seems odd, odio hand it makes perfect sense!"

  • (cs) in reply to JAPH
    JAPH:
    In Java, a class cannot be defined in multiple files. The first file with a class with that name and package is used; any other classes are ignored. Of course, if your application uses multiple ClassLoaders, each class loader could have a different class.

    I think that we're just circling back to the fact that TRWTF is Java.

  • Franky (unregistered)

    TRWTF is obviously that the whole company seems to be pissed that they only have one machine, yet noone thought about just reimaging the second one from scratch or to an image made from the first one ...

  • anonymous (unregistered) in reply to Franky
    Franky:
    TRWTF is obviously that the whole company seems to be pissed that they only have one machine, yet noone thought about just reimaging the second one from scratch or to an image made from the first one ...
    Obviously there were a handful of mission-critical processes that ran on the server, so they couldn't just nuke it and start over.

    What, you thought that it was a test server?

  • Anomaly (unregistered)

    The real question everyone should be asking is if this is exploitable or not >=D

    And everyone complaining about why the system was still around, do you all have exactly one machine dedicated to each individual task that needs being done at your job? Didn't think so. Its perfectly reasonable that the machine did all of its other functions CORRECTLY. And its just with the java devs that the machine was flawed.

  • I. (unregistered) in reply to Steve The Cynic

    Not really, no. I suspected something like that halfway through the story. They're not that strange, really.

  • Charles F. (unregistered) in reply to mihi
    mihi:
    Jars in there will not only override all user classes, but even classes in the JRE itself (rt.jar), and even if the classes have been dumped into a memory-mappable image (classes.jsa) for "class data sharing", the classes in the image are ignored in case you have a class with same name in jre/lib/endorsed.
    That's interesting; I did not know that much detail about endorsed.

    I think you have a point that the doc I linked to should mention the "endorsed" directory at least to tell people to leave it alone. (Some warning of platform instability seems appropriate.)

  • Franky (unregistered) in reply to Anomaly
    Anomaly:
    do you all have exactly one machine dedicated to each individual task that needs being done at your job?
    pretty much, yes. on prod everything has at least its own VM if not dedicated hardware and on dev/test at max 2-3 services per machine and a pretty great deploy system -> just export the services list from the machine, nuke the thing, reimport, done. And in the process of also porting those to VMs -> snapshots ftw!
  • Charles F. (unregistered) in reply to chubertdev
    chubertdev:
    Charles F.:
    That's just the way it's always been done, and you should never expect it to get better. I've known exactly how Java works since birth.

    is pretty much what I'm getting

    This is why we can't have nice things.

  • TRWTF (unregistered)
    article:
    “I already checked timestamps and file sizes,” Andrew said.
    Yeah, you da man!

    I write awesome shit, and I don't know why my customer complain because it all compile.

    Does chksum not work for JAR files? Is filesize and timestamp really all that relaible, I mean, really?

  • Joey (unregistered) in reply to Tintin
    Tintin:
    The real WTF here is that this is apparently an IT company, they have a PC sat somewhere which no one dare touch, and NO ONE thought to just reformat and reinstall everything (or even just throw it out, if it's THAT cursed) - instead they have an idle PC sitting there wasting space.

    Really?

    It would have been much quicker to reformat and reinstall than all the time that lots of people had spent trying to get it working.

    Seems relatively normal for most of the places I've owrked at....

    We don't dare blat it, because it might be useful, but it doesn't work and everyone's too scared ot try to fix it.

  • maisr (unregistered) in reply to Steve The Cynic
    Steve The Cynic:
    Charles F.:
    Steve The Cynic:
    The rules are arcane because they follow an unobvious schema.
    ...unless you read the fucking documentation.
    Steve The Cynic:
    It should, at the very least, be located *after* CLASSPATH in the search order, if for no other reason than to allow me to substitute my own version of a class file for debugging purposes
    1. You do not understand what a platform extension is if you think it's something a user application should be able to override.
    [b[Well, then the WTF is that the platform extension directory is writeable by *anything* except installers.[/b] And also that some hooligan at the WTF Inc shown here put a version of their own code in the platform extension directory.
    Charles F.:
    2. You do not understand Java debugging. It's 2013 and we don't debug our code by putting "print" statements in it. There is no need for "debugging" versions of jars. Jars can be compiled without debugging information, but this is rarely done anymore as it adds relatively little to the class size. Also, unless you have the source, you are stuck with the jar you have.
    I don't debug my code by putting print statements in it either, most of the time (debugging packet flows in live OS kernels is a different matter). Interactive debuggers are a superb invention, one that dates back to the 1980s and even earlier. (So they are an older invention than Java.)

    However, in the case given, the schmucks were apparently reduced to doing something dangerously close to debug-prints in order to have a clue what was going on.

    Charles F.:
    3. If you want, you can remove the libraries from the "ext" directories on your workstation. You should not do this on any other environment, nor should libraries be put there unless it is part of a well-understood configuration.
    And in the article, we saw a good demonstration of why they should not be there. There's a case to be made that perhaps the classes loaded from the platform extension directories should be forced to be members of a particular package hierarchy otherwise they won't be loaded. Notably: if it's NOT part of a platform extension package hierarchy, WTF is it doing there? Why would we *want* to load out-of-hierarchy packages from there?

    Knowing the way that these things work, I'd put small amounts of money on there not being a suitable set of package hierarchies available for such a whitelist.

    yeah, it would be a thorough pain if that pesky "root" guy came in and had the ability to like change stuff...

    unfortunately, you can't beat stupid, and there's no point trying to cater for stupid, because it just encourages more stupidity (I think it took MS win95 to learn that)

  • Toby (unregistered) in reply to faoileag
    faoileag:
    no laughing matter:
    From the HTML comments of the article: It was a sysadmin who left the company well before this happened. Why would a sysadmin do such changes without instructions from the developers?
    Perhaps the sysadmin knew about the class loading chain in java and did this as a deliberate prank?

    After all, accidentally copying a file to the wrong directory can happen to anybody; it would be hard to prove intent.

    It's also possible for someone with escalated rights (ie an admin) to accidentally copy something to the wrong place...

    Eunics is very trusting (which is good, because I am smarter than the computer) but that means when I'm in a directory (which perhaps I shouldn't be in) and say: cp <somefile.jar> . Eunics doesn't do the win thing and say "Are you certain you have even the foggiest idea of what is goin on here?...Reaaly?", instead it mutters to itself something about "...just because you thinks you know what you is doing better than me, master..." and does what I told it to.

    Then later in their home directory the admin has a moment where they go "I'm sure I copied that file here, where did the little sucker go" and end up not finding it and recopying it. Meanwhile they've left said file in a place where it is going to cause Confusion and Delay.

  • Duc (unregistered) in reply to Toby

    Obviously, these guys never work with a Websphere Application Server or any Java EE Application Server for that matter. Either set the class loader policy to parent last or dump the class loader list to see the order of class loading would've solved the problem within half an hour.

  • AliBaba (unregistered)

    If it was so bad that no one was even going to use the server, why not just rebuild it? I know 'just format it' a sin in the Ops world but it's better than the server just sitting there collecting dust and wasting electricity and holding up dev resources.

  • (cs) in reply to ANON
    ANON:
    Steve The Cynic:
    ANON:
    Nice story, but actually I see no WTF.
    You don't think that arcane rules about how classes get loaded are sufficiently WTFy for you? Think, man, think! The fact that there can be magical invisible objects that silently override the ones you put there explicitly is a WTF of the highest order.

    Actually this one made me think of an Elegant Syntax Error.

    So you'd say the WTF is how Java works? Each platform has its own dependency hell. Java didn't find a perfect solution, but neither did anyone else, so how can you blame it?

    Every large system or language environment contains WTFWTT (WTF Were They Thinking) features that no one sane would use. These are the features that someone presents verbally, during some architectural meeting beginning with, "Wouldn't it be cool if..." (This is also known as a, "Seemed OK at the time," moment.)

    Let us consider a gross example: Altered GO TO in COBOL. This wonderful language element allows you to silently change the destination of a GO TO statement, that says "GO TO ALPHA", so that it really jumps to BETA.

    If GO TO is "considered harmful", working on a program built around altered GO TO is like being consigned to the hall of the damned. (I know from personal experience: I've been in that hall. No, I didn't create the hall; I just wanted to take the creator somewhere and beat him to death with the claw end of a claw hammer.)

    Not only is altered GO TO a WTFWTT feature, only the truly insane would consider actually using it these days.

    Yet there are always "language lawyers" who will look at every jot and tittle of the definition of a system or language and learn all these WTFWTT features, and say to themselves, "Well, that's a cool feature...how can I use that?" The result is things like "mockingbird"...or as I already noted, the hall of the damned.

    To remain among the sane, grasshopper, you must learn to identify and avoid the WTFWTT features. You must learn to say to yourself, "I suppose that feature might have some use, but it is best not to use it. Ever." It might be a more mundane existence, but at least you won't be looked on as creator of the hall of the damned.

  • yeah sure (unregistered) in reply to KattMan
    I would challenge even Mr. expert here to solve this problem quickly without going down the same route as the people in the story. Why, because he knows how it is supposed to work and what you are not supposed to do. You don't check for stupid user tricks first, you run it, see a problem and start looking at things that might be the problem.

    Not claiming to be "Mr. expert" here, but "dodgy JVM" was the first thing I thought of at the first reference of "cursed". Even though it was more about some arcane JVM bug, for there are many, rather than someone stupid enough to spoil the lib/ext directory.

    So my course of action would be "grab the latest JRE*; export JAVA_HOME; $JAVA_HOME/bin/java genCore.jar". Whoosh, problem solved!

      • as in "the latest JRE within currently used major version"
  • (cs) in reply to Doodpants
    Doodpants:
    “No, it’s not.” Christine, pale, looked to him out of the corner of her eye. “Who put that jar file there in the first place?”

    Obviously, it was put there by the president's sick daughter. "Sick" as in "sick and twisted".

    Nah, it was probably Hanzo.

  • Your Naem (unregistered)

    You fool! The CEO's e-mail application requires the old genCorejdbc.jar in order to parse XML from an access database stored on the network share! You stole his internet and this is affecting productivity!

  • (cs)

    These are not the JAR files that you're looking for.

    “Who put that jar file there in the first place?”
    In all likelihood, the NSA.

Leave a comment on “The Curse of the Warped Bootstrap”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article