- Feature Articles
- CodeSOD
- Error'd
-
Forums
-
Other Articles
- Random Article
- Other Series
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
In other words, the trillion dollar mistakes is to not use NULLs lol
Admin
And this is exactly why NULL wasn't a billion dollar mistake; sometimes the best answer is to crash and not to return a "safe" but incorrect value
Admin
Indeed. Null is returned because of an error. This is not only a perfectly reasonable choice; it is a documented one. Posix requires the null return in case of error. Using an asterisk for no-password-here is also a long-standing, well-known feature.
As usual, the real trillion dollar mistake is Python-level programmers deciding that they know better how to write safe programs than the Standards committees.
Admin
Thanks for the writeup; knew about "the SSH bug" but not about the details, and this strengthens my irrational dislike of musl[a].
Semi-rhetorical question: If musl returned '!' (disabled login), would the bug have been elided?
[a]: Not denying that its design choices have their place (particularly on containers/fully constrained images), but for general use/places where dynamic linking isn't taboo it ain't it.
Admin
Exactly. It's also on all the devs of the programs that crashed instead of just simply checking that return value for NULL; it's such unbelievably lazyness that's then catered to. That's truly TRWTF.
Admin
This elides some details. In particular the broken code does not return "" if the hash begins with ''. Instead it returns '!', to ensure a failure can never match the hash.
The vulnerability comes from openssh passing in a hash from a different account when the original account does not have a password (as a timing side-channel mitigation).
Thus, musl crypt gets a hash NOT starting with '', so returns "" for a too-long input.
OpenSSH then compares that to the original not-hash "*", rather than the hash it passed to crypt (as crypt assumes), so they match.
Admin
Well, markdown destroyed that. Assume empty quotes contain a single asterisk...
Admin
It did! Or tried to. It interacted badly with openssh's attempts to mitigate timing side-channel attacks.
See my other mangled comment for details.
Admin
Funny thing is that even if you're using glibc, you won't always see crypt(3) return a null on errors. Distributions like Debian and Arch are based on glibc, but use Björn Esser's libxcrypt instead of GNU's libcrypt and it tries to address the same one that the musl developers did -- That there's a lot of code out there that calls crypt(), doesn't check the return value and then crashes because of the null pointer.
"Upon error, crypt_rn and crypt_ra return a null pointer. crypt_r and crypt may also return a null pointer, or they may return a pointer to the invalid hash, depending on how libcrypt was configured."
It's not going to return '' as an invalid hash, but it's not guaranteed to be NULL either. This kind of boundary-crossing confusion is what lead to the whole sshhole issue in the first place -- The OpenSSH team wrote their code assuming it would always be using the standard BSD libcrypt, or at least differing to PAM. Musl assumes that crypt() will only be called on hashes that make sense and not just for time-filling busy work. And somewhere around 1970 Bill Joy assumed that '' couldn't possibly match a valid password hash. It's just that nobody was able to look at every component at once and see that they were interacting in ways that weren't quite right.
Admin
Close. OpenSSH only borrows the algo+salt of another account's hash when hashing the input password. This avoids the timing side channel by hashing a password for a nonexistent user the same way as existing users. It ends up hashing SALT+TOOLONGPASSWORD and triggering the asterisk response, which matches the asterisk in shadow. crypt() should have returned '!' for all failures, as a ''!' in shadow means that account cannot login via any method. If OpenSSH sees '!' it won't even try to authenticate the account, so there is no chance it would compare the '!' from crypt() to the '!' in shadow.
Admin
In this case, has sshd forked off yet? If not a NULL pointer crash would take down the root sshd, turning this into a DoS attack.
Admin
Unfortunately OpenSSH was not quite paranoid enough. They expected the crypt to follow the POSIX (Open Group) definition. I don't think this is that unreasonable. I am personally unmoved by the arguments that musl libc made the right choice to avoid crashing poorly written software.
Admin
Took me a while, but I finally got the "<asterisk> SSH hole" joke....
Admin
I get the feeling that, while returning "NULL" is a poor choice of how to indicate failure, returning a star -- or any other single char -- is insider obfuscation at its worst. Reminds me of co-workersXXX co-idiots who thought returning , say, 1e33, from a function which returns numeric was a good error indicator. Anyway, I hereby propose replacing the star (or anus if you wish) character in this case with the string " SSH';) DROP TABLE passwords ; -- "
Admin
I inherited code that has a function that looks up an attribute on a record. If no record is found based on the criteria specified by the function arguments, it returns the string "FAKE". If the function encounters an error, it returns the string "error". The attribute being looked up is unlikely to be, but very well could be, either "FAKE" or "error".
Admin
BTW, the only way this function is ever used is to check for existence of a record by comparing the return value to "FAKE". There is a very similarly named function that fetches the same attribute, but has less weird failure behavior that used when the code actually wants the value of the attribute.
Admin
Perhaps some Unix/Linux admin can answer this question. Is there anybody in the World who does not make sure the following is in their sshd config file?
I get that the vulnerability is still a huge problem even with that, but surely...
Admin
But most Linux systems can be rooted once you have access to a shell and a C compiler. I don't expect that to keep serious hackers out. Drive-by script kiddies, perhaps.
Admin
The reason for the bug is because crypt() does NOT have a standard return value on error. In fact, the NetBSD page on it details the history of return values - https://man.netbsd.org/crypt.3
Basically, it was supposed to always return something, returning NULL was never an option because the initial implementations of crypt() could never fail. Now they take a complex string with parameters and now there's a need to return errors, and there's no way to return errors. Returning NULL isn't an option as many programs assumed it never failed because it never had a reason to fail. So there are a bunch of different ways to indicate an error. NetBSD probably did it best by returning a string shorter than 13 characters.
But musl is also correct for returning "*" as well. It's the end result of a function that shouldn't fail suddenly needing to fail.
Admin
Beside the variety of crypt implementations, why does the password hash check not special handle * ?? Or is that a "clever" trick of the developers to automatically fail a hash test (which only works when you're given a valid hash to begin with)?
Basically, when two dumb decisions meet (TM)
Admin
The real WTF is making it fail because the input was "too long". Hashing a long string has the same space/storage requirements as hashing a short one.
If you're paranoid about somebody stealing CPU cycles then you could make it so it only hashes the first 256 characters or something like that (256 char passwords wouldn't be less secure than 257 char passwords), but it shouldn't FAIL.
Admin
Well, the silver lining is that AI isn't making these problems - they were already there. If we weather through this period, we'll be left with a healthier codebase and maybe some lessons. If we don't, then the crash was inevitable either way, and we kinda deserved it.
Admin
It just kills the sshd, which is annoying, but much less serious than hijacking your server. A proper health monitoring will see the dying daemon, restart it and report the issue. Then you can figure out, why it's crashing, investigate the bug and fix it.
NULL is no problem at all. It is a sane and sensible choice to do certain things. If your app crashes because of NULLs, then the solution is not to put NULL checks everywhere, but to figure out, where the NULL comes from and apply proper handling at that point. (Unless NULL is a valid value, than you have to handle it properly everywhere, obviously.)
Spoiler alert! Returning a special "normal" value, which you have to inspect to see, whether a problem happened is the same thing as returning NULL, which you have to check to see whether a problem happened. Except that it doesn't crash your app and keeps running, giving you at best problems later at place for away from the cause or at worst giving you root shell.
Admin
hard to be left with a healthier codebase when people are going around with their LLMs, putting ten times as many vulnerabilities in code they don’t fully understand
Admin
"I am personally unmoved by the arguments that musl libc made the right choice to avoid crashing poorly written software."
I tend to agree. I am also very much reminded of the System32/Syswow64 nonsense that Microsoft came up with instead of using System64 for 64-bit and System32 for 32-bit. What we got was just a really bad way to accommodate older/existing software, there biggest reasoning being because of hardcoded paths like for system32 and such.
This was rubbish because 1) existing software would need to edited and rebuilt to make a native 64-bit version any how and would be expected to update things based on docs, 2) new software targeting 64-bit would also be expected to read and apply correct ways of finding locations and such, and 3) existing 32-bit software that had hardcoded locations were already looking for system32, so that didn't even need to be touched.
They could have just added special handling for looking drivers, which were 64-bit and thus, if they went with system64, would live there but "WINDOWS\system32\drivers" could be redirected just like they did the other way around in our reality, as an example.
I'm all for backwards compatibility, but it needs to be done sensibly, not half-assedly.
Admin
I'm surprised that it wasn't the original null return value that caused the vulnerability. I could easily see a null hash being taken to mean "this account has no password - come on in!"
Admin
Hm. Yes, point taken.
Admin
In that case, NetBSD is not Posix-compliant, either.
Admin
No way musl choice of * is an honest mistake. It is a planted vulnerability.