• akozakie (unregistered)

    Correct reasoning, but not much will happen longterm. Remember the leftpad disaster?

  • Happy Coder (unregistered)

    and there are also configuration options which can disable this behavior in older versions Actually, there is an exploit that can circumvent the config parameter now https://nvd.nist.gov/vuln/detail/CVE-2021-45046

  • (nodebb)

    Much of the world silently uses Open Source software as part of doing business, often at no cost.

    "at no cost" is incorrect. There is always a cost, even if it isn't an up-front monetary cost.

  • (nodebb)

    I've written a few string template processing methods over the years. Never, not once, did I think, hey, you know what would be a good idea? Parse the placeholder as an expression which can make network requests. That's not enough - process the responses and if it's a Java class, execute the code. And the worst, do all this as part of logging.

    Next on TDWTF: GM recalls a billion cars because if somebody pours olive oil in the fuel filler, the car decides it should cook the occupants. Hey, olive oil is healthy, isn't it?

  • Robin (unregistered)

    Thanks for this. I may be misunderstanding, but as I see it, TRWTFs are:

    • log4j apparently has deliberately allowed remote code execution as a feature for years, and seemingly no-one stopped to think if this was a bad idea or not
    • despite this, no-one ever thought to exploit it until what, last week

    I'm not a Java guy, so I really hope I'm missing something - but neither of the above make any sense to me...

  • (nodebb) in reply to Mr. TA

    I have no idea how this bug came to be, but I suspect that Log4j is simply reusing some other Java built-in component for string formatting. And flexible formatting options for a built-in all-purpose string formatter isn't bad. What is bad is passing untrusted strings to the formatter. But when was the last time you expected your string formatter to have an RCE capability? I think this is one of those cases where each individual component in isolation looks "fair enough", but put them together and you've accidentally gone critical and cooked up a nuclear weapon.

  • (nodebb) in reply to Robin

    I think it's entirely possible a small group of hackers knew it for years and used it without being detected, and also that the larger mass of professionals didn't even think it was possible for something like this to exist. I, for one, do not expect my car to have occupant cooking feature, nor one which is triggered by the presence of olive oil.

  • (nodebb) in reply to Mr. TA

    Never, not once, did I think, hey, you know what would be a good idea?

    That's not really what happened. One guy said "Hey let's allow people to do things like look up user names in LDAP via JNDI". That's the rationale for adding JNDI to Log4J. See it in action here: https://issues.apache.org/jira/browse/LOG4J2-313

    Another guy said "I'd like to store code in LDAP so it can be stored in a central and redundant place". This sort of thing happened a lot in the early days of LDAP - like most technologies, several "visionaries" saw it as the next big thing and it should be used for everything.

    Add those two things together, subtract a security analysis, and you get Log4Shell.

  • (nodebb)

    Sigh. This article is completely wrong in one details: the scope is not even remotely that widespread as it makes it. Yes, log4j is used almost everywhere, but log4j is NOT affected by this exploit at all, because it does NOT support any jndi interpolation (or any other interpolation, other that plain string - basically %s, but with a little WTF syntax "{}").

    The whole stupid interpolation thing is present in log4j2, which is (despite its name) a completely different library. Yes, it tries to promote itself as a new version of log4j (major version upgrade), but very few people actually falls for that (although it's mostly because of the "I've been using the same stuff for 10 years and I'm going to continue until retirement" attitude). Of course, "few" here is still a lot.

  • (nodebb) in reply to Vilx

    I never had the need to have anything in the placeholder other than a field reference. I always make sure to support navigation with dots. In the email and SMS template processing, it supports expressions but those are very limited, basically math operators and ternary conditions.

    I'd love to see one example where string formatting requires making network requests of any sort.

  • (nodebb)

    Also, I can find a quote from a BlackHat presentation in 2016: "Applications should not perform JNDI lookups with untrusted data" (https://www.blackhat.com/docs/us-16/materials/us-16-Munoz-A-Journey-From-JNDI-LDAP-Manipulation-To-RCE.pdf). I'm sure that's not the first time the advice has been given, but it's evidence that the security guidance was out there in about the same timeframe that this bug was introduced. It's good advice and the reason LOG4J2-313 (the request to add JNDI support to Log2j) should have been rejected.

  • (nodebb) in reply to Jaime

    Exactly, that's a terrible thought process. Why would you make network requests from a logging library? Logger should output the information it has, not go on an information hunt around the system. What if there are system wide problems, and logger tries to write an error? It tries getting data using LDAP requests, those fail, and now that failure is also logged.

    No no no. Bad.

    Addendum 2021-12-15 07:56: PS. If, for whatever reason, somebody's username is needed in the log entry, the code which calls the log method should get that data and then pass it in as a name value dictionary.

  • Hal (unregistered) in reply to Robin

    Its more like someone thought it was a good idea to have some expression evaluation in logging and it probably is - so you don't have t be assembling strings constantly in your own code. I am not a Java guy either but being able to send something like "${method}:${Worker.id}: An ${e.name} exception was raised" is likely useful. Now obviously that is trivial and might as well just be evaluated in the exception handler or whatever method body is logging but more complex formulations where you want to include values that might not be in the current scope is where this becomes useful. So I think its wrong to condemn it out of hand. The failure of course was in the sandboxing, if you going to do something like this you need to make darn sure that the scope of values and objects that are accessible are remain well known - but of course patterns like DI run almost directly counter to that...

    I don't think its really a code quality issue, or a closed open source, who is paying issue. I think its when you turn something that should be somewhat simple like a log sink into a big complex highly configurable one size fits all 'solution' surprise surprise emergent behavior happens. micro services are fine example - sure maybe you only wrote 400 lines of really understandable code but its calling into mLOCs of stuff you only think you know how any of works. Now think hard about PaaS stuff on AWS/Azure/Google realize how how much complexity there has been presented as simple to the developer and now consider the blast radius there, and how you can't just role a workaround config change if you are Amazon/Microsoft because you don't know what you will be breaking.

    Going to be fun times ahead.

  • LCrawford (unregistered) in reply to Vilx

    | I have no idea how this bug came to be, but I suspect that Log4j is simply reusing some other Java built-in component for string formatting. And flexible formatting options for a built-in all-purpose string formatter isn't bad. What is bad is passing untrusted strings to the formatter

    This - it's the equivalent of calling fprintf(logfile, log_line) , where log_line is user controlled input. While the C code probably can't download and execute remote code, it is easy to crash and possibly exploit controlled code execution.

  • kdgregory (unregistered) in reply to Mr. TA

    I'd love to see one example where string formatting requires making network requests of any sort.

    The string interpolation feature of Log4J 2.x is primarily intended for configuration. Where it's very reasonable to retrieve data from a central source (maybe less useful to execute code from that source, but I'm sure there's someone who will defend it).

    The issue happens because the Log4J 2.x PatternLayout also does string interpolation. I've been fairly deep in the Log4J 2.x code, and never knew this. I suspect that if you'd asked 10,000 Log4J 2.x users about this a week ago, maybe one or two would have said they knew about it. Maybe one actually uses it.

  • (nodebb) in reply to Kamil Podlesak
    Sigh. This article is completely wrong in one details: the scope is not even remotely that widespread as it makes it. Yes, log4j is used almost everywhere, but log4j is NOT affected by this exploit at all, because it does NOT support any jndi interpolation (or any other interpolation, other that plain string - basically %s, but with a little WTF syntax "{}").

    The whole stupid interpolation thing is present in log4j2, which is (despite its name) a completely different library.

    You're wrong on one detail: log4j 2.x is not a "completely different library", implying that it's written and maintained by somebody else. It's still the logging library from Apache. Yes, it is a different major version, but it was released in 2014, so any new or significantly-updated projects from the past 7 years are most likely using 2.x. There are certainly a number of organizations that never upgraded from 1.x, but you can't act like it's some obscure fork of log4j that only a few people use.

  • David Green (unregistered)

    I still think a lot of companies like to use open source, but don't want to contribute back. The last team I managed were looking to enhance a library we were using on GitHub. We were going to write wrappers in our own code, but we really wanted to make the changes directly and contribute back. However, there was concern about legal liabilities (unfounded or otherwise) or the company "giving away its IP." I honestly don't know how to reconcile this.

  • (nodebb) in reply to kdgregory

    Still, if configuration settings are useful info, the code which calls log method should get that config and give it to logger as a dictionary of some sort. This is a prime example of Swiss army knife anti pattern, logs sometimes need to show current configuration, therefore let's tie configuration retrieval into logging. NO!! That's bad. You know, separation of concerns and all that.

  • my name is missing (unregistered)

    Another fine example of "You can pay me now, or pay the ransom later". My last employer spent eternities examining the license agreements of open source, but no one ever looked at the project itself.

  • kdgregory (unregistered) in reply to my name is missing

    no one ever looked at the project itself

    You can't.

    Here's the source code for Log4J2: https://gitbox.apache.org/repos/asf?p=logging-log4j2.git can you find the vulnerability?

    Does it help if I give you the hint that StrSubstitutor is what performs substitutions?

    The current trunk for log4j-core has 89,778 lines of code. That doesn't include add-ons such as log4j-kafka or log4j-flume-ng.

    And Log4J is a tiny part of a typical Java deployment.

  • Hasseman (unregistered)

    The child of Bobby Tables, Little Jindi Code

  • (nodebb) in reply to kdgregory

    The issue happens because the Log4J 2.x PatternLayout also does string interpolation. I've been fairly deep in the Log4J 2.x code, and never knew this. I suspect that if you'd asked 10,000 Log4J 2.x users about this a week ago, maybe one or two would have said they knew about it. Maybe one actually uses it.

    Having interpolation happen on the core logging templates would be a bit nasty, but those tend to be pretty tightly controlled within the application and so wouldn't be a problem. Having interpolation be done on the per message template (the first argument to log.info() etc) would be a much less nice thing — you've got active code right next to this that can do the needful so why would you want a second round? — but the real problem here is that the interpolation was being done on the result after substitution in of the argument values, an approach that lets an attacker give you a value that you log “safely”, and in so doing still have the attack go ahead. And it means that the logger is double-processing the resulting log message, which slows things down (for bonus “fun”).

    I also find JNDI's approach to loading code from LDAP to be Very Bad, and it makes me wonder what sorts of other problems are down there. Most applications don't need to load code from arbitrary places; they know exactly where they are going to get code from at startup. Directory services most certainly shouldn't be doing an end-run around that sort of policy.

  • (nodebb) in reply to kdgregory

    can you find the vulnerability?

    Code block starting at:

    https://github.com/apache/logging-log4j2/blob/be881e503e14b267fb8a8f94b6d15eddba7ed8c4/log4j-core/src/main/java/org/apache/logging/log4j/core/pattern/MessagePatternConverter.java#L126

    Ironically, commented with “TODO can we optimize this?”. Yes, Virginia, we can optimize it by deleting it.

  • kdgregory (unregistered) in reply to dkf

    Code block starting at:

    I'm guessing you've already looked at the code to be able to respond that quickly. And you know that there is a vulnerability.

    Would you have called out that code before knowing that the vulnerability exists?

    Do you regularly inspect code at that level before adding a dependency to your application? (and if you do, are you selling that as a service?)

  • Lot of half-truths here (unregistered)

    1 - Vulnerable log4j2 versions are 2.9 (since 2013) - 2.50 (this week) + some Java 8+ that has JNDI unsecured. Versions 1.x don't have this bug, and of course alternative implementations dont; newer java versions have JNDI secured; 2 - the bug is: substitute parameters to message template and THEN substitute the variables -- obviously wrong behavior; user cannot log values like "${java:os}" as a result; 3 - Chinese Woonsan Ko wrote and then maintainer Ralph Goers approved the code to include jndi lookup which was not really justified. Not even local JNDI context lookups; do explicit log4j context map and copy the value there.

  • a robot (unregistered)

    OK I'd agree that allowing callouts to untrusted servers is a very bad thing, but surely TRWTF is executable binary payloads. I thought the idea of downloading binary code from a server (even a server you trust) and just executing it had disappeared by the mid-90's

  • Splatmandeux (unregistered)

    "Given the scope of the flaw, however, not only will there be extended fallout from it, but it also forces the tech community to confront the relationship between businesses and OpenSource. "

    I'm don't think the relationship is much different than the one between any other 3rd party provider, opensource or otherwise. If log4j was a commercial offering would that have mitigated the issue? What if the provider was out of business? What if they had in fact fixed it, but you didn't purchase the update agreement? What if you did purchase the update agreement but never got around to deploying the update?

    A lot of business world is built upon technology not "grown at home", and for good reason. Rolling your own functionality is ridiculously costly. A company that relies on outside vendors (including opensource) needs to have a role within the company to stay on top of the developments at those vendors and the clout to make the necessary changes happen. Unfortunately that role is usually double-booked onto someone low-level with another time consuming role.

  • John (unregistered)

    The problem actually is that the enabled JNDI lookup was the default, only (originally) disabled by a reverse-negative option.

    I don't see what the payment method has to do with the problem. Paid-for code and developers are equally likely to have bad designs.

  • Splatmandeux (unregistered) in reply to Hal

    Logging is one of those things that seem to always get kitchen-sinked. Every company I have worked for has rolled their own logging library, or at least a veneer of one over top of something like log4cplus/log4j. They all became complex messes over time with inner-platform effect the rule. It's mostly self-inflicted because most of the extended functionality almost never gets used. It's one of the most ridiculous domains in software development I've ever seen. Like catnip for architecture astronauts.

  • (nodebb) in reply to kdgregory

    The current trunk for log4j-core has 89,778 lines of code.

    How does a logging framework get to ninety K lines of code?

    That is the question that people should be asking. The fact that some of those lines of code were self-evidently written by someone who was less intelligent than a cabbage is neither here nor there.

  • Rob (unregistered) in reply to Dragnslcr

    While Log4J 2 is probably used a lot, it's still not as bad as a lot of people claim (but still bad!). That's because it's not the only logging framework there is.

    Spring Boot includes the Log4J 2 API, but only to redirect its calls to SLF4J to Logback. That's a fork (or something like that) from Log4J 1, which does not have this JNDI security hole. Quarkus and JBoss EAP (among others) use jboss-logging, which is another logging framework that also doesn't have this JNDI security hole. Tomcat uses something called JULI which in turn uses java.util.logging. Guess what - it doesn't have this JNDI security hole.

    In all of these cases this JNDI security hole is not present as long as you don't include Log4J 2 through some other dependency.

  • (nodebb) in reply to a robot

    TRWTF is executable binary payloads.

    Are you serious? We all do that all the time. Only today I downloaded an executable binary payload and then ran it: it was the Apple Xcode developer suite. Every time I download an executable or library from a server online or an app store I am downloading an executable binary payload. TRWTF is not downloading executable binary payloads, but having a mechanism that allows other untrusted people to download executable binary payloads onto your computer.

  • xtal256 (unregistered)

    "We break up our large, complicated application into a bunch of tiny services, and focus on how messages travel between these services."

    Fun fact, this is actually what Object Oriented Programming was supposed to be all about - sending messages between small "things" (objects). That is until it - like everything else in software development - got so complicated that people now hate it.

  • (nodebb) in reply to Rob

    I'm amazed ANYONE uses log4j2, I thought everyone moved to slf4j about 8 years ago. Any library that doesn't use slf4j these days is just making life hard for its users.

  • Worf (unregistered)

    Well, I like the explanation, which is extremely clear and concise.

    And I can see why it happens - log4j lets you log more than the standard "Error in FILE(LINE)::func : Error Message Here" type strings (and it bugs me to no end that it's func and not FUNC - surely whoever added it to C99 knew there were OCD people out there? Or did they add ways to change function names or something?)

    Now you might want to add "Permission Denied: {USER} did not have permission to do {ACTION}." and even do lookups of what that user could do "{USER} Permissions {PERMISSION_LIST}" which would involve LDAP lookups.

  • löchleindeluxe (unregistered) in reply to Jeremy Pereira

    Eh, I have vague memories of at least one case of "Company X let the update domain on $cloudService lapse so now legitimate-but-EOL software could pull untrusted updates". I think it even was X=Microsoft.

  • (nodebb) in reply to Vilx

    Nope. It was a deliberate decision. apache.org, Log4j 2 JIRA entry, 17th July, 2013: "JNDI Lookup plugin support"

  • SG (unregistered) in reply to yaytay

    I'm amazed ANYONE uses log4j2, I thought everyone moved to slf4j about 8 years ago. Any library that doesn't use slf4j these days is just making life hard for its users.

    So, two things on that. First, SLF4J doesn't log anything on its own - it's just a wrapper around other logging frameworks such as Logj4j2, and as such, will likely have the same vulnerabilities as the underlying libraries.

    Second, SLF4J has been kind of static since it was written... the API pre-dates Java 8, so it's unable to take advantage of modern tricks like passing suppliers for lazy-fetching computationally-expensive values... something that Log4J2 does well. There's an SLF4J 2 in the pipeline which improves that, but it's been about seven years since Java 8 hit the scenes, and SLF4J still doesn't have a stable release which can take advantage of it. A lot of people have moved on...

  • Philip (unregistered) in reply to Jaime

    To make it even worse: the underlying problem was reported in 2014: https://issues.apache.org/jira/browse/LOG4J2-905

    With the question being asked:

    In addition to this option, will you add a global option to disable lookups in messages ? i.e. in log4j configuration and/or via system property ?

    Of course, the only reasonable answer is:

    We have no plans to do that as this should give you exactly what you need.

    ... and here we are.

  • (nodebb)

    I've done a lot of digging and found design issues, documentation issues and feature interactions at play. But I didn't see anything I'd really call a 'bug'; everything was basically working as designed. And yeah, that's a scary thought.

    My write-up is here: https://jedwidz.hashnode.dev/log4j-vulnerability-what-the-faq-happened

Leave a comment on “Don't Lookup: The Log4j Debacle”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article