• (nodebb)

    In other words, the trillion dollar mistakes is to not use NULLs lol

  • Tim (unregistered)

    And this is exactly why NULL wasn't a billion dollar mistake; sometimes the best answer is to crash and not to return a "safe" but incorrect value

  • RLB (unregistered) in reply to MaxiTB

    Indeed. Null is returned because of an error. This is not only a perfectly reasonable choice; it is a documented one. Posix requires the null return in case of error. Using an asterisk for no-password-here is also a long-standing, well-known feature.

    As usual, the real trillion dollar mistake is Python-level programmers deciding that they know better how to write safe programs than the Standards committees.

  • docontra (unregistered)

    Thanks for the writeup; knew about "the SSH bug" but not about the details, and this strengthens my irrational dislike of musl[a].

    Semi-rhetorical question: If musl returned '!' (disabled login), would the bug have been elided?

    [a]: Not denying that its design choices have their place (particularly on containers/fully constrained images), but for general use/places where dynamic linking isn't taboo it ain't it.

  • (nodebb) in reply to RLB

    Exactly. It's also on all the devs of the programs that crashed instead of just simply checking that return value for NULL; it's such unbelievably lazyness that's then catered to. That's truly TRWTF.

  • David-T (unregistered)

    This elides some details. In particular the broken code does not return "" if the hash begins with ''. Instead it returns '!', to ensure a failure can never match the hash.

    The vulnerability comes from openssh passing in a hash from a different account when the original account does not have a password (as a timing side-channel mitigation).

    Thus, musl crypt gets a hash NOT starting with '', so returns "" for a too-long input.

    OpenSSH then compares that to the original not-hash "*", rather than the hash it passed to crypt (as crypt assumes), so they match.

  • David-T (unregistered) in reply to David-T

    Well, markdown destroyed that. Assume empty quotes contain a single asterisk...

  • David-T (unregistered) in reply to docontra

    It did! Or tried to. It interacted badly with openssh's attempts to mitigate timing side-channel attacks.

    See my other mangled comment for details.

  • (nodebb)

    Funny thing is that even if you're using glibc, you won't always see crypt(3) return a null on errors. Distributions like Debian and Arch are based on glibc, but use Björn Esser's libxcrypt instead of GNU's libcrypt and it tries to address the same one that the musl developers did -- That there's a lot of code out there that calls crypt(), doesn't check the return value and then crashes because of the null pointer.

    "Upon error, crypt_rn and crypt_ra return a null pointer. crypt_r and crypt may also return a null pointer, or they may return a pointer to the invalid hash, depending on how libcrypt was configured."

    It's not going to return '' as an invalid hash, but it's not guaranteed to be NULL either. This kind of boundary-crossing confusion is what lead to the whole sshhole issue in the first place -- The OpenSSH team wrote their code assuming it would always be using the standard BSD libcrypt, or at least differing to PAM. Musl assumes that crypt() will only be called on hashes that make sense and not just for time-filling busy work. And somewhere around 1970 Bill Joy assumed that '' couldn't possibly match a valid password hash. It's just that nobody was able to look at every component at once and see that they were interacting in ways that weren't quite right.

  • (nodebb) in reply to David-T

    Close. OpenSSH only borrows the algo+salt of another account's hash when hashing the input password. This avoids the timing side channel by hashing a password for a nonexistent user the same way as existing users. It ends up hashing SALT+TOOLONGPASSWORD and triggering the asterisk response, which matches the asterisk in shadow. crypt() should have returned '!' for all failures, as a ''!' in shadow means that account cannot login via any method. If OpenSSH sees '!' it won't even try to authenticate the account, so there is no chance it would compare the '!' from crypt() to the '!' in shadow.

  • (nodebb) in reply to Tim

    In this case, has sshd forked off yet? If not a NULL pointer crash would take down the root sshd, turning this into a DoS attack.

  • Troy Roberts (unregistered)

    Unfortunately OpenSSH was not quite paranoid enough. They expected the crypt to follow the POSIX (Open Group) definition. I don't think this is that unreasonable. I am personally unmoved by the arguments that musl libc made the right choice to avoid crashing poorly written software.

  • Hmmmm 🤨 (unregistered)

    Took me a while, but I finally got the "<asterisk> SSH hole" joke....

  • (nodebb)

    I get the feeling that, while returning "NULL" is a poor choice of how to indicate failure, returning a star -- or any other single char -- is insider obfuscation at its worst. Reminds me of co-workersXXX co-idiots who thought returning , say, 1e33, from a function which returns numeric was a good error indicator. Anyway, I hereby propose replacing the star (or anus if you wish) character in this case with the string " SSH';) DROP TABLE passwords ; -- "

  • (nodebb) in reply to cellocgw

    returning a star -- or any other single char -- is insider obfuscation at its worst

    I inherited code that has a function that looks up an attribute on a record. If no record is found based on the criteria specified by the function arguments, it returns the string "FAKE". If the function encounters an error, it returns the string "error". The attribute being looked up is unlikely to be, but very well could be, either "FAKE" or "error".

  • (nodebb)

    BTW, the only way this function is ever used is to check for existence of a record by comparing the return value to "FAKE". There is a very similarly named function that fetches the same attribute, but has less weird failure behavior that used when the code actually wants the value of the attribute.

  • (nodebb)

    Frequently, you might make accounts like root nominally inaccessible by giving it no password, so this is a big deal.

    Perhaps some Unix/Linux admin can answer this question. Is there anybody in the World who does not make sure the following is in their sshd config file?

    PermitRootLogin no
    

    I get that the vulnerability is still a huge problem even with that, but surely...

  • Hanzito (unregistered) in reply to jeremypnet

    But most Linux systems can be rooted once you have access to a shell and a C compiler. I don't expect that to keep serious hackers out. Drive-by script kiddies, perhaps.

  • (nodebb)

    The reason for the bug is because crypt() does NOT have a standard return value on error. In fact, the NetBSD page on it details the history of return values - https://man.netbsd.org/crypt.3

    The behavior of crypt() on errors isn't well standardized. Some implementations simply can't fail (unless the process dies, in which case they obviously can't return), others return NULL or a fixed string. Most implementations don't set errno, but some do. Version 2 of the Single UNIX Specification (SUSv2'') specifies only returning NULL and setting errno as a valid behavior, and defines only one possible error (ENOSYS,The functionality is not supported on this implementation.'') Unfortunately, most existing applications aren't prepared to handle NULL returns from crypt(). The description below corresponds to this implementation of crypt() only. The behavior may change to match standards, other implementations or existing applications.

    crypt() may only fail (and return) when passed an invalid or unsupported setting, in which case it returns a pointer to a magic string that is shorter than 13 characters and is guaranteed to differ from setting. This behavior is safe for older applications which assume that crypt() can't fail, when both setting new passwords and authenticating against existing password hashes.

    Basically, it was supposed to always return something, returning NULL was never an option because the initial implementations of crypt() could never fail. Now they take a complex string with parameters and now there's a need to return errors, and there's no way to return errors. Returning NULL isn't an option as many programs assumed it never failed because it never had a reason to fail. So there are a bunch of different ways to indicate an error. NetBSD probably did it best by returning a string shorter than 13 characters.

    But musl is also correct for returning "*" as well. It's the end result of a function that shouldn't fail suddenly needing to fail.

  • Your Name (unregistered)

    Beside the variety of crypt implementations, why does the password hash check not special handle * ?? Or is that a "clever" trick of the developers to automatically fail a hash test (which only works when you're given a valid hash to begin with)?

    Basically, when two dumb decisions meet (TM)

  • FTB (unregistered)

    The real WTF is making it fail because the input was "too long". Hashing a long string has the same space/storage requirements as hashing a short one.

    If you're paranoid about somebody stealing CPU cycles then you could make it so it only hashes the first 256 characters or something like that (256 char passwords wouldn't be less secure than 257 char passwords), but it shouldn't FAIL.

  • (nodebb)

    Well, the silver lining is that AI isn't making these problems - they were already there. If we weather through this period, we'll be left with a healthier codebase and maybe some lessons. If we don't, then the crash was inevitable either way, and we kinda deserved it.

  • Kotarak (unregistered) in reply to Coyote

    In this case, has sshd forked off yet? If not a NULL pointer crash would take down the root sshd, turning this into a DoS attack.

    It just kills the sshd, which is annoying, but much less serious than hijacking your server. A proper health monitoring will see the dying daemon, restart it and report the issue. Then you can figure out, why it's crashing, investigate the bug and fix it.

    NULL is no problem at all. It is a sane and sensible choice to do certain things. If your app crashes because of NULLs, then the solution is not to put NULL checks everywhere, but to figure out, where the NULL comes from and apply proper handling at that point. (Unless NULL is a valid value, than you have to handle it properly everywhere, obviously.)

    Spoiler alert! Returning a special "normal" value, which you have to inspect to see, whether a problem happened is the same thing as returning NULL, which you have to check to see whether a problem happened. Except that it doesn't crash your app and keeps running, giving you at best problems later at place for away from the cause or at worst giving you root shell.

  • no (unregistered) in reply to Allexxann

    hard to be left with a healthier codebase when people are going around with their LLMs, putting ten times as many vulnerabilities in code they don’t fully understand

  • Terry Merop (unregistered) in reply to Troy Roberts

    "I am personally unmoved by the arguments that musl libc made the right choice to avoid crashing poorly written software."

    I tend to agree. I am also very much reminded of the System32/Syswow64 nonsense that Microsoft came up with instead of using System64 for 64-bit and System32 for 32-bit. What we got was just a really bad way to accommodate older/existing software, there biggest reasoning being because of hardcoded paths like for system32 and such.

    This was rubbish because 1) existing software would need to edited and rebuilt to make a native 64-bit version any how and would be expected to update things based on docs, 2) new software targeting 64-bit would also be expected to read and apply correct ways of finding locations and such, and 3) existing 32-bit software that had hardcoded locations were already looking for system32, so that didn't even need to be touched.

    They could have just added special handling for looking drivers, which were 64-bit and thus, if they went with system64, would live there but "WINDOWS\system32\drivers" could be redirected just like they did the other way around in our reality, as an example.

    I'm all for backwards compatibility, but it needs to be done sensibly, not half-assedly.

  • (nodebb)

    I'm surprised that it wasn't the original null return value that caused the vulnerability. I could easily see a null hash being taken to mean "this account has no password - come on in!"

  • (nodebb) in reply to no

    Hm. Yes, point taken.

  • RLB (unregistered) in reply to Worf

    In that case, NetBSD is not Posix-compliant, either.

  • CommanderJavik (unregistered)

    No way musl choice of * is an honest mistake. It is a planted vulnerability.

Leave a comment on “In the Hole”

Log In or post as a guest

Replying to comment #706179:

« Return to Article