• MuTaTeD (unregistered)

    Am I the Frist Sysadmin?

  • KV (unregistered)

    Frist.jar and First.jar differ!

  • AP² (unregistered)

    Rails is convention over configuration.

    The JVM is configuration over configuration.

  • DonRobo (unregistered)

    TRWTF is that he didn't check the extension folder the second he saw that the jar he copied over wasn't the jar that was used.

  • Fritz, a.k.a. Fritzo (unregistered)

    So the WTF was that Andrew a.k.a. Andrewo didn't know the basics of what he was working with?

  • RFoxmich (unregistered)

    “No, it’s not.” Christine, pale, looked to him out of the corner of her eye. “Who put that jar file there in the first place?”

    I did muhuhuhuhahahahahah!!! ;-)

  • ANON (unregistered)

    Nice story, but actually I see no WTF.

  • 30into (unregistered)

    TRWTF is I spent the whole story waiting for a really bad "To Kill A mockingbird" pun...

  • (cs) in reply to ANON
    ANON:
    Nice story, but actually I see no WTF.
    You don't think that arcane rules about how classes get loaded are sufficiently WTFy for you? Think, man, think! The fact that there can be magical invisible objects that silently override the ones you put there explicitly is a WTF of the highest order.

    Actually this one made me think of an Elegant Syntax Error.

  • Jo (unregistered)

    The WTF is that there's a lib/ext directory. And that somebody placed a jar there.

    If you do application programming and use standard deployment practices, you never see lib/ext, and don't get too aware of it as a potential source of classpath problems. So it's a nice catch actually.

    The "who placed the jar there" bit was overly dramatic though - probably just somebody who didn't know proper deployment procedures. If WTFs were caused by demons or anything, they could be healed by an exorcism, problem solved... it's human error and stupidity we're combatting. (Friedrich Schiller: "Against stupidity the gods themselves struggle in vain.")

  • anonymous_coder() (unregistered)

    Am I the only one who thought "after a certain amount of time, reimage the SOB and press on regardless" would be the way to go? Every once in a while, I've managed to bung up a dev server to the point that it won't work anymore, and a reinstall of the OS was in order...

  • toot (unregistered)
    it's demons
    signs-you-might-be-a-bad-coder.html
  • (cs) in reply to anonymous_coder()
    anonymous_coder():
    Am I the only one who thought "after a certain amount of time, reimage the SOB and press on regardless" would be the way to go?
    You are not the only one, that was also my first thought--in fact, it's the FIRST thing I would have done. Based on the story set up, the thing wasn't a server, it was a space heater.
  • (cs) in reply to DonRobo
    DonRobo:
    TRWTF is that he didn't check the extension folder the second he saw that the jar he copied over wasn't the jar that was used.
    I don't think that enumerating all the possible locations and searching them is the best option. Use 
    -verbose:class
    and get the VM to tell you where it's finding the class.
  • ANON (unregistered) in reply to Steve The Cynic
    Steve The Cynic:
    ANON:
    Nice story, but actually I see no WTF.
    You don't think that arcane rules about how classes get loaded are sufficiently WTFy for you? Think, man, think! The fact that there can be magical invisible objects that silently override the ones you put there explicitly is a WTF of the highest order.

    Actually this one made me think of an Elegant Syntax Error.

    So you'd say the WTF is how Java works? Each platform has its own dependency hell. Java didn't find a perfect solution, but neither did anyone else, so how can you blame it?

  • Amy (unregistered)

    File access monitors are very, very useful. I spent around a minute using one to figure out an issue a coworker had spent more than 3 days on.

    (And on Windows, registry monitors too.)

  • ZoomST (unregistered) in reply to RFoxmich
    RFoxmich:
    > “No, it’s not.” Christine, pale, looked to him out of the corner of her eye. “Who put that jar file there in the first place?”

    I did muhuhuhuhahahahahah!!! ;-)

    Ooooh! Spooky! Just ideal for Halloween, I think. Maybe it can be improved: “Who put that jar full of human blood there in the first place?” Trigger or threat? :)

  • faoileag (unregistered) in reply to Steve The Cynic
    Steve The Cynic:
    Actually this one made me think of an Elegant Syntax Error
    Nice link, I somehow hadn't read that one before. Reminds me of a time when a colleague and me thought about creating a C++ header file that would provide latin versions of all C++ key words. We never did it, though.

    Hmmm, now I wonder if one could patch the java compiler to replace "while" with something else...

  • faoileag (unregistered) in reply to ZoomST
    ZoomST:
    Maybe it can be improved: “Who put that jar full of human blood there in the first place?
    Doesn't Stan owe Andrew two beers for fixing genCore or mockingbird? A Bucket of Blood would be perfect in the run-up to Halloween, I think ;-)
  • (cs) in reply to Jo
    Jo:
    (Friedrich Schiller: "Against stupidity the gods themselves struggle in vain.")

    Also a great Isaac Asimov Novel....

  • Charles F. (unregistered) in reply to Steve The Cynic
    Steve The Cynic:
    You don't think that arcane rules about how classes get loaded are sufficiently WTFy for you?
    The rules are actually very straightforward. They are only "arcane" if you don't understand them.

    The people in the story didn't check lib/ext because JDBC libraries are not the kind of thing you'd put in there. It's like the classic tech support story where it's discovered -- after a series of fixes are tried -- that the computer is not working because power is out at the user location. The whole 20-minute support call could have been much shorter if the tech had asked, "Is your building power working?" but since it should be obvious that you need power for your workstation to function, you only get to that possibility after exhausting all of the more sensible ones.

    CAPTCHA: validus "Putting JDBC libraries in lib/ext is not a validus thing to do."

  • (cs) in reply to Charles F.
    Charles F.:
    Steve The Cynic:
    You don't think that arcane rules about how classes get loaded are sufficiently WTFy for you?
    The rules are actually very straightforward. They are only "arcane" if you don't understand them.

    The people in the story didn't check lib/ext because JDBC libraries are not the kind of thing you'd put in there. It's like the classic tech support story where it's discovered -- after a series of fixes are tried -- that the computer is not working because power is out at the user location. The whole 20-minute support call could have been much shorter if the tech had asked, "Is your building power working?" but since it should be obvious that you need power for your workstation to function, you only get to that possibility after exhausting all of the more sensible ones.

    CAPTCHA: validus "Putting JDBC libraries in lib/ext is not a validus thing to do."

    The rules are arcane because they follow an unobvious schema. The most obvious artefact of the class loading is the CLASSPATH, and lib/ext overrides that invisibly. It should, at the very least, be located after CLASSPATH in the search order, if for no other reason than to allow me to substitute my own version of a class file for debugging purposes, without breaking the way that it works elsewhere on the machine. (Sure, I can explicitly put the class/jar file on the command line, and maybe I should, but the question then is, "Does that override lib/ext?" and once you start to go down that road, you realise it's paved with caltrops.)

  • (cs) in reply to ANON
    ANON:
    So you'd say the WTF is how Java works? Each platform has its own dependency hell. Java didn't find a perfect solution, but neither did anyone else, so how can you blame it?
    The WTF in this story is that Java works like that. Compare that to, for example, C/C++ include file loading rules. Sure, they have their oddities (*), but the most important thing is that all compilers I've ever used load files from the system directory *after* having tried everywhere else (INCLUDE env variable, -I directives, etc.) rather than *before*.

    (*) The principal oddity is the behaviour of #include "file.h". This loads from the current directory, but which current directory? The answer is, "the directory that contains the file doing the include," which makes it hard to override file.h with your own version.

  • I R BOFH (unregistered)
    “No, it’s not.” Christine, pale, looked to him out of the corner of her eye. “Who put that jar file there in the first place?”

    A deafening peal of thunder answered her nervous question.

    Thor? Zeus? The Thunder King?

    I don't get it.

  • dna (unregistered)

    I'm just an hotliner, not a developper. but i'm asking myself : did this problem would have survived to the helpdesk approch : desinstall java (delete directory) and reinstall java.

  • Fred (unregistered)

    TRWTF is Java

  • Charles F. (unregistered) in reply to Steve The Cynic
    Steve The Cynic:
    The rules are arcane because they follow an unobvious schema.
    ...unless you read the fucking documentation.
    Steve The Cynic:
    It should, at the very least, be located *after* CLASSPATH in the search order, if for no other reason than to allow me to substitute my own version of a class file for debugging purposes
    1. You do not understand what a platform extension is if you think it's something a user application should be able to override.

    1. You do not understand Java debugging. It's 2013 and we don't debug our code by putting "print" statements in it. There is no need for "debugging" versions of jars. Jars can be compiled without debugging information, but this is rarely done anymore as it adds relatively little to the class size. Also, unless you have the source, you are stuck with the jar you have.

    2. If you want, you can remove the libraries from the "ext" directories on your workstation. You should not do this on any other environment, nor should libraries be put there unless it is part of a well-understood configuration.

    CAPTCHA: augue It's hard to augue with an idiot.

  • Fritz, a.k.a. Fritzo (unregistered) in reply to ZoomST
    ZoomST:
    RFoxmich:
    > “No, it’s not.” Christine, pale, looked to him out of the corner of her eye. “Who put that jar file there in the first place?”

    I did muhuhuhuhahahahahah!!! ;-)

    Ooooh! Spooky! Just ideal for Halloween, I think. Maybe it can be improved: “Who put that jar full of human blood there in the first place?” Trigger or threat? :)

    Well there's one person who's pretty infamous for putting jars where they don't belong. I.e. George Lucas (a.k.a. Georgo Lucaso)

  • Charles F. (unregistered) in reply to Steve The Cynic
    Steve The Cynic:
    The WTF in this story is that Java works like that. Compare that to, for example, C/C++ include file loading rules.
    The WTF in your comment is that you think that Java class loading can be compared in any way to C/C++ include files.

    CAPTCHA: suscipit I suscipit that your were put in cryostasis in 1991. Now, just recently thawed out, you are confused by our strange modern technologies.

  • Krunt (unregistered) in reply to Charles F.
    Charles F.:
    Jars can be compiled without debugging information, but this is rarely done anymore as it adds relatively little to the class size.

    How does the inclusion of debug information affect performance of a Jar though (if at all)?

    The reason I ask is that it does have an impact on a .NET assembly, and most of us here do not release binaries to production unless they've been compiled to... well, Release mode.

  • Mandalf the Mainly (unregistered)

    Demons you say?

    /sigh

    Let me put on my robe and wizard's hat...

  • (cs) in reply to dna
    dna:
    I'm just an hotliner, not a developper. but i'm asking myself : did this problem would have survived to the helpdesk approch : desinstall java (delete directory) and reinstall java.

    probably not I'm afraid the uninstaller may keep the directory alive so a reinstall will not need to be reconfigured

  • (cs) in reply to Krunt
    Krunt:
    Charles F.:
    Jars can be compiled without debugging information, but this is rarely done anymore as it adds relatively little to the class size.

    How does the inclusion of debug information affect performance of a Jar though (if at all)?

    The reason I ask is that it does have an impact on a .NET assembly, and most of us here do not release binaries to production unless they've been compiled to... well, Release mode.

    there is no difference between java compiled for debug or release (there is no optimizer in javac)

    a jvm running in release mode (aka not in debug mode) will simply ignore the debug info

  • faoileag (unregistered) in reply to Charles F.
    Charles F.:
    Steve The Cynic:
    The rules are arcane because they follow an unobvious schema.
    ...unless you read the fucking documentation.
    Having done that, am I right to assume that... ... I, the application developer, do not have any control over what .jar files might exist in jre/lib/ext/? ... the .jar files in jre/lib/ext/ are loaded prior to my application's .jar files, establishing a method-lookup-up precedence chain? ... it is therefore at least theoretically possible that I pick a name for one of my application's .jar files that also names a .jar file in the jre/lib/ext/ directory? ... and that, to stretch the thought experiment even further, both my and the extension's .jar file could export the same methode (e.g. init())? ... meaning my init() method would have no chance of ever getting called leaving me going quietly insane trying to find out why my application does not work as intended?

    If the answer to all these questions is "yes", I'll definitely will have to keep that in mind! :-)

  • Tintin (unregistered)

    The real WTF here is that this is apparently an IT company, they have a PC sat somewhere which no one dare touch, and NO ONE thought to just reformat and reinstall everything (or even just throw it out, if it's THAT cursed) - instead they have an idle PC sitting there wasting space.

    Really?

    It would have been much quicker to reformat and reinstall than all the time that lots of people had spent trying to get it working.

  • Charles F. (unregistered) in reply to Krunt
    Krunt:
    How does the inclusion of debug information affect performance of a Jar though (if at all)?
    To add to what ratchet freak said, compiling with debug information merely includes some optional information in the class file.

    The most visible is the "line number table" that maps positions in the byte code to lines in the source code. This is necessary for source-level debuggers to work. The other visible one is the "local variable name" table which records the source-code name for items on the local variables stack. This is also relied on by source-level debuggers. Neither of these tables adds a lot of bulk to the class file, so unless you are running in a low-memory environment, it's not a problem to include them. Some people strip them off as part of an attempt to obfuscate their code.

    If you don't have the line number table in your class, you don't get line numbers in your stack traces which makes debugging more fun.

  • Doodpants (unregistered) in reply to Charles F.

    “No, it’s not.” Christine, pale, looked to him out of the corner of her eye. “Who put that jar file there in the first place?”

    Obviously, it was put there by the president's sick daughter. "Sick" as in "sick and twisted".

  • Charles F. (unregistered) in reply to faoileag
    faoileag:
    I, the application developer, do not have any control over what .jar files might exist in jre/lib/ext/?
    Yes.
    faoileag:
    ... the .jar files in jre/lib/ext/ are loaded prior to my application's .jar files, establishing a method-lookup-up precedence chain?
    Yes,
    faoileag:
    ... it is therefore at least theoretically possible that I pick a name for one of my application's .jar files that also names a .jar file in the jre/lib/ext/ directory?
    Yes, but in practice, no.

    Java programs don't really care about the names of the JAR files, but the names of the classes in the JAR files. It is theoretically possible that you could name one of your classes to be the same as one in a higher-precedence JAR file, but this would require some really bad naming of your classes.

    Class files are assigned a "package" which is really just a namespace. Your classes are really referred to by their "fully qualified name." If you have an application class called 

    Date
    it is probably in a package like 
    com.initech.accounting.brokenapp
    So, your actually FQN class name is 
    com.initech.accounting.brokenapp.Date
    There are likely many "Date" objects available, but they have FQNs like 
    java.util.Date
    or 
    com.lonelyhearts.meeting.Date
    or whatever.

    By convention, your package name starts with your domain name in reverse (except for core classes that start with "java" or "javax"). This means that class files from different organization already start out in distinct namespaces. Does that clear it up?

  • (cs)

    You don't even need to know where someone might have dropped the devil's jar. That's why we have search tools.

  • (cs) in reply to faoileag
    faoileag:
    Having done that, am I right to assume that... ... I, the application developer, do not have any control over what .jar files might exist in jre/lib/ext/? ... the .jar files in jre/lib/ext/ are loaded prior to my application's .jar files, establishing a method-lookup-up precedence chain? ... it is therefore at least theoretically possible that I pick a name for one of my application's .jar files that also names a .jar file in the jre/lib/ext/ directory? ... and that, to stretch the thought experiment even further, both my and the extension's .jar file could export the same methode (e.g. init())? ... meaning my init() method would have no chance of ever getting called leaving me going quietly insane trying to find out why my application does not work as intended?

    If the answer to all these questions is "yes", I'll definitely will have to keep that in mind! :-)

    You're correct that you have no control over what .jar files might exist in lib/ext, and that the extensions have priority over the application (although you can hack around it by using command-line options to add your jar to the boot classpath, which has even higher priority. That's a very ugly hack, though).

    However, the key thing which protects developers' sanity is that the .jar file name is irrelevant: what matters are the fully qualified names of the classes, so provided that you follow naming conventions and no-one does something stupid like the configuration in the article, you don't need to worry about clashes.

  • faoileag (unregistered) in reply to Charles F.
    Charles F.:
    Does that clear it up?
    Well, yes, it did! Thanks!
  • faoileag (unregistered) in reply to pjt33
    pjt33:
    ...that the .jar file name is irrelevant: what matters are the fully qualified names of the classes, so provided that you follow naming conventions and no-one does something stupid like the configuration in the article, you don't need to worry about clashes.
    Thanks as well!
  • (cs)

    I love it, the real WTF is us.

    ok who cares how the java bootstrap and such works, who cares if sanely naming your files makes things so you don't have this problem. Who cares if Mr. expert above knows every step and sane way to handle things so it all just simply works.

    I would challenge even Mr. expert here to solve this problem quickly without going down the same route as the people in the story. Why, because he knows how it is supposed to work and what you are not supposed to do. You don't check for stupid user tricks first, you run it, see a problem and start looking at things that might be the problem.

    Yes it would all work, naming doesn't clash etc, until some STUPID user copies your companies file into the extensions folder and now any changes to that file no longer get used and things just seem to break.

    You expect things to work, you expect the people you work with to know what they are doing, until something like this happens and you get stumped for a bit. The story here had at least one expert that understood and followed a logical path to resolution. The problem is not something you would think would happen because "NO SANE JAVA DEVELOPER WOULD DO THAT!" Hence why it is arcane, hidden, something only the wise men would think of, and even they won't think of it first. Anyone that says they would see the problem and immediatly say, "we got a bad extension file" is a liar and should never write another line of code again, hell, he should probably have his hands cut off just to make sure, and his eyes put out so he can't see to critisize any other code.

    Tamam Shud!

  • Valued Service (unregistered)

    I think people are missing the point that a application part was counted as an extension to the platform.

    That's the problem.

    It's fair that platform extensions take priority, but to have a application domain level be considered a platform extension, when it isn't really a part of the platform at all, is the issue.

    Now, whether you want to bother to "fix" that, is another story. The "fix" for now seems to be for people to not put parts of their application in the ext and use common sense.

  • (cs) in reply to Fred
    Fred:
    TRWTF is Java
    There are many good reasons to think this, but the extension mechanism of todays' story certainly isn't.

    Todays' story is not a WTF, it's a RTFM!

  • (cs) in reply to Mandalf the Mainly
    Mandalf the Mainly:
    Let me put on my robe and wizard's hat...
    Aight! Finally this forum is going somewhere.
  • (cs)

    From the HTML comments of the article: It was a sysadmin who left the company well before this happened. Why would a sysadmin do such changes without instructions from the developers?

    Obviously some people really need rigid change and configuration management policies.

    And be shot when violating those!

  • (cs) in reply to Charles F.
    Charles F.:
    Steve The Cynic:
    The rules are arcane because they follow an unobvious schema.
    ...unless you read the fucking documentation.
    Steve The Cynic:
    It should, at the very least, be located *after* CLASSPATH in the search order, if for no other reason than to allow me to substitute my own version of a class file for debugging purposes
    1. You do not understand what a platform extension is if you think it's something a user application should be able to override.
    Well, then the WTF is that the platform extension directory is writeable by *anything* except installers. And also that some hooligan at the WTF Inc shown here put a version of their own code in the platform extension directory.
    Charles F.:
    2. You do not understand Java debugging. It's 2013 and we don't debug our code by putting "print" statements in it. There is no need for "debugging" versions of jars. Jars can be compiled without debugging information, but this is rarely done anymore as it adds relatively little to the class size. Also, unless you have the source, you are stuck with the jar you have.
    I don't debug my code by putting print statements in it either, most of the time (debugging packet flows in live OS kernels is a different matter). Interactive debuggers are a superb invention, one that dates back to the 1980s and even earlier. (So they are an older invention than Java.)

    However, in the case given, the schmucks were apparently reduced to doing something dangerously close to debug-prints in order to have a clue what was going on.

    Charles F.:
    3. If you want, you can remove the libraries from the "ext" directories on your workstation. You should not do this on any other environment, nor should libraries be put there unless it is part of a well-understood configuration.
    And in the article, we saw a good demonstration of why they should not be there. There's a case to be made that perhaps the classes loaded from the platform extension directories should be forced to be members of a particular package hierarchy otherwise they won't be loaded. Notably: if it's NOT part of a platform extension package hierarchy, WTF is it doing there? Why would we *want* to load out-of-hierarchy packages from there?

    Knowing the way that these things work, I'd put small amounts of money on there not being a suitable set of package hierarchies available for such a whitelist.

  • (cs)

    With all those statements about worsening weather, I am thinking the next thing that happens is an extensive test of all the back-up power systems in the facility.

  • (cs) in reply to Charles F.
    Charles F.:
    The WTF in your comment is that you think that Java class loading can be compared in any way to C/C++ include files.
    You think they don't both involve searching for an object down a list of file system resources? (directories, .jar files, etc.)

    Notably, the chaos that can be caused by incorrect ordering of those resources is very comparable.

    I think you can also make a case that you might want to be able to activate some sort of output from the JVM that shows the class-loading decisions it made when loading each class (aside from dumb built-in things like strings). That would have shown up the issue immediately, because their classes would have been shown loading from the wrong location.

    For all I know, such a thing exists...

Leave a comment on “The Curse of the Warped Bootstrap”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article