One Asterisk, Ten Years: Alpine Linux's Silent SSH Bug

Here is a number that should bother you: 257. For roughly ten years, typing any password of 257 characters or more into an SSH prompt on the wrong kind of Linux system didn't get you rejected — it got you in. Any password. Keyboard mashing, a base64 dump of your vacation photos, half a Slack message you pasted by mistake: as long as it was long enough, the door swung open. The Alpine Linux SSH bug disclosed as "sshole" is one of those vulnerabilities that reads like a prank until you check the conditions, at which point it reads like an incident report.

The short version: on OpenSSH built against musl libc — that's Alpine, OpenWrt, Buildroot, Yocto, and a staggering share of the Docker base images on the internet — accounts that were supposed to be mathematically impossible to log into would accept any password between 257 and 1,024 characters. Root. Your key-only deploy user. The default alpine user on Alpine's official AWS and Azure images, which ships with passwordless doas. The fix landed in OpenSSH 10.6p2. But the story of how it happened is a chain of small, sensible decisions stretching back to 1980 — and every single link in it looked correct at the time.

How the Alpine Linux SSH Bug Actually Works

To understand the collision you need three pieces, each owned by a different codebase and a different decade. First: how Linux stores passwords. When you set a password, the hash lands in /etc/shadow, one line per account. A line for a normal account looks like $6$salt$hash, where $6$ means sha512crypt. But accounts that should never accept a password — daemons, key-only users, locked root — get a placeholder instead:

alice:$6$yJ3kXw9q$k2P9vLm4...:19723:0:99999:7:::
deploy:*:19723:0:99999:7:::
root:*:19723:0:99999:7:::

When you log in, sshd takes what you typed, hashes it with the stored salt, and string-compares the result against the stored field. Same string, you're in. The asterisk is a sentinel: it isn't the hash of anything, so no password can ever match it. That has been the locking mechanism for key-only accounts since the early days of Unix. It works because the comparison can never succeed — unless the hashing function itself can be made to produce an asterisk. Which brings us to piece two.

Piece two is musl's error handling. Traditionally, crypt() returns NULL when it can't do its job — say the salt requests an algorithm the system doesn't have. The trouble is that decades of software never checked for NULL and would cheerfully pass it to strcmp and segfault. So in 2012, musl made a deliberate design choice: its crypt() would never return NULL. On failure it would return a string guaranteed not to match any real password hash, so broken callers would see a wrong password instead of crashing. The failure string they picked was *. Three weeks later musl added sha512crypt with a 256-byte input cap — hashing cost scales with password length, so giant passwords are a denial-of-service vector. Reject anything longer, which is a failure, which returns *. The comment documenting this sat in the source the entire time.

Piece three is sshd being polite. In 2016, CVE-2016-6210 showed you could enumerate valid usernames by timing responses: nonexistent users got an instant rejection because there was no hash to compute. The fix was elegant — for accounts with no usable hash, sshd borrows another account's salt and hashes the supplied password anyway, purely to burn the same amount of time. A slower no. The catch: this path doesn't just trigger for nonexistent users. It also triggers for real users whose shadow field is *, because an asterisk isn't a usable hash. So now sshd computes crypt(your_password, borrowed_salt) and compares the result against *.

/* what sshd effectively does in this configuration */
const char *stored = shadow_field(user);        /* "*" for locked accounts */
const char *salt   = pick_salt(user);           /* borrowed: "*" isn't a valid hash */
char *ret = crypt(typed_password, salt);        /* musl: "*" if password > 256 bytes */
if (ret && strcmp(ret, stored) == 0)
allow_login();                              /* "*" == "*"  →  welcome in */

Every element has to align: the stored field is *, crypt fails, and the failure value is also *. On glibc, a failed crypt returns NULL, which fails the comparison safely. On libxcrypt systems the failure string is *0 — one character different, and that one character is the whole ballgame, because *0 never equals *. On musl, the error value and the lock value are the same character, and strcmp has no way to tell an error from a match. Two meanings, one wire, and the receiver can't disambiguate.

A Decade of Perfectly Reasonable Decisions

What makes this bug worth studying isn't the mechanism — it's that nobody involved did anything obviously stupid. Start in 1980: back when password hashes sat in world-readable /etc/passwd, Bill Joy created an account called daemon that nobody should ever log into, and put a * where its password hash would go. A clean sentinel, decades ahead of its time. (The real-name field reportedly read "the devil himself.") Unix still locks accounts exactly this way.

Musl's 2012 decision looks defensible from every angle too. Crashing callers are a real problem, and a C library that weaponizes NULL against decades of sloppy application code makes a lot of enemies. Returning a can-never-match string is graceful degradation. So is capping password length to stop hash-based denial of service. So is sshd's timing fix — padding out responses for unhashable users is standard practice. So, even, is what Alpine did next. In 2019, CVE-2019-5021 revealed Alpine's Docker images shipped with an empty root password field — in some configurations, no password at all. Alpine locked root with !. But without PAM, sshd treats ! as fully locked, which kills SSH key logins too — and cloud images live and die by key logins. So in 2022, Alpine 3.16 moved root to *: still locked, keys work, everyone happy. Except * was precisely the string musl's crypt could be tricked into printing.

Every step was a defensible response to a real problem. That's what makes this uncomfortable — you could have been in the room for each decision and nodded along.

The near-misses are almost worse. A Debian developer once asked glibc to adopt musl's exact * behavior; six years later Debian moved to libxcrypt, whose *0 closed the hole by accident. A BusyBox thread complained about passwd choking on a 257-character password — same asterisk, same limit, and nobody in the thread said "sshd." A security reviewer patched sshd's pick_salt() function with the actual bug sitting four lines below it, four days before someone finally found it. This thing was walked past, repeatedly, by people looking almost directly at it.

Is Your Infrastructure Affected by the musl OpenSSH Bug?

The saving grace of sshole is that it needs a full house of conditions before it pays out. Miss any one and you're safe. You are exposed if all of these are true:

• Your sshd is built against musl: Alpine, OpenWrt, Buildroot, Yocto, or a Docker image descended from alpine with openssh-server installed.
• sshd is not going through PAM — Alpine's openssh-server doesn't by default, or you've set UsePAM no. PAM's code path never calls crypt for a * account.
• Password authentication is enabled. It is by default.
• At least one account has a bare * in /etc/shadow: key-only users, LDAP-backed accounts, root on Alpine.
• At least one other account has a real password hash, so there's a salt to borrow.

Meet all five, and every * account on the box accepts any password from 257 to 1,024 characters. The nightmare configuration is Alpine's official AWS and Azure images: the default alpine user is key-only (so, *) and has passwordless doas. A long-password login there isn't privilege escalation — it's a front door straight to root. One mitigation deserves credit for keeping this quiet for so long: OpenSSH's PermitRootLogin has refused password logins for root since 2015, so the most obvious target was partially shielded by default. Nothing shielded the deploy users and service accounts.

If you're on Debian, Ubuntu, Fedora or anything else glibc-based, you're fine. Anything routing through PAM: fine. Dropbear: fine. OpenBSD: fine. One caution for the skeptics — early community attempts to reproduce this have been inconsistent, because the five conditions are easy to miss one at a time. Someone tries the official Alpine image, doesn't realize root's field is ! rather than *, fails, and moves on. Treat "couldn't reproduce on my laptop" as no evidence at all about production. Check the conditions, not the vibes.

The Real Root Cause: In-Band Signaling and Magic Strings

It's tempting to file sshole under "weird libc edge case" and move on. The failure mode is older and more general than that: in-band signaling. The character * was drafted into two jobs — in /etc/shadow it means "no password exists for this account," and in musl's crypt it means "hashing failed." Two messages sharing one channel, with no way for the receiver to tell them apart. Sentinel values embedded in ordinary data are how systems end up exploiting themselves. NULL was an acceptable error signal (crash-prone callers notwithstanding). An exception is better. A Result<T, E> that forces the caller to handle failure explicitly is best. The practical rule, the one I'd tattoo on every systems programmer: no magic strings. If you absolutely must have one, pick something that cannot collide with any value in the problem domain — libxcrypt's *0 beats musl's * by exactly one byte, and that byte is the difference between a quirk and a CVE.

There's a second lesson here about silent compatibility shims. Musl's graceful-* decision exists to stop broken software from crashing — which means it stops broken software from ever being noticed as broken. A segfault is ugly, but it's also a signal: someone files a bug, someone fixes the caller. Swallow the error and the mistake compounds quietly for a decade. I'm increasingly convinced this is the deepest divide in systems design: engineering that resists catastrophe tends to fail loudly and early, while engineering that optimizes for smoothness accumulates hidden debt until the bill arrives all at once.

And it should retire a comforting myth: that old, widely deployed code is more trustworthy because it's battle-tested. Age doesn't distribute scrutiny evenly — it concentrates scrutiny on the hot paths and leaves the cold ones to fossilize. Everyone audits the crypto. Almost nobody audits the boring glue where sshd talks to libc about passwords, because that code "has worked forever." The intersection of two minority configurations — musl systems, and within those the ones not using PAM — is exactly the kind of place a bug can live undisturbed for ten years.

How to Fix the sshole Bug and Check Your Logs

The patch is the easy part: OpenSSH 10.6p2 closes the hole. On Alpine, that's one command. If you can't update today, the mitigation is equally blunt and completely effective — turn password authentication off. No password check, no bug. Frankly you should do both, because key-only auth was the right answer before this bug existed and will remain so after everyone has patched. It echoes a lesson we've seen in other patch-versus-mitigate debates: when the mitigation is cheap and total, don't get clever.

# patch: openssh 10.6p2 or later
apk upgrade openssh
# can't patch yet? passwords off, keys only
echo "PasswordAuthentication no" > /etc/ssh/sshd_config.d/00-no-passwords.conf
rc-service sshd restart

Then the harder question: did anyone get there first? Ten years is a long dwell time. What you're looking for is successful password authentications for accounts that never had passwords — key-only deploy users, service accounts, LDAP identities, root on Alpine boxes. Successful logins for those accounts via the password method should be vanishingly rare; any hit is a five-alarm fire. The uncomfortable truth is that sshd logs the success, not the password length, so a lot of the direct evidence evaporates on write. And as we've covered before, cloud authentication logging has blind spots even on a good day — instance-level sshd logs on short-lived VMs are exactly where they live. Grep what you have. Assume it's incomplete.

What Ten Quiet Years Should Teach Us

The discovery itself deserves as much attention as the bug. The researcher who found sshole didn't stare at the crypto — they stared at the boring boundary where OpenSSH talks to libc, built sshd in every configuration they could think of (glibc, musl, PAM, no PAM, shadow, no shadow), and threw roughly 17,600 malformed logins at the matrix. Weird hashes, broken accounts, passwords far too long. Every build agreed on every login except one: musl without PAM. That one said yes to things it shouldn't. That's not a lucky find; that's coverage. Differential testing across configuration matrices is how you find the bugs that hide in intersections — and it's exactly the kind of exhaustive, tireless work that agent swarms turn out to be disturbingly good at. The next sshole will not get ten quiet years.

The open question that should keep operators up at night is whether ten years really passed with nobody noticing — or whether the wrong people noticed. Consider the accidental case alone: paste the wrong clipboard contents into a password prompt — half a Slack thread, an entire SSH private key, anything over 257 characters — and on a vulnerable box, a locked account simply opens. If you've never fumbled a paste into a password field, you're a more careful person than anyone I've worked with. Somebody, somewhere, has logged into a box they didn't mean to log into and quietly closed the terminal. Maybe many somebodies.

So patch, yes. Turn passwords off, absolutely. But then take an hour and go hunting in your own systems for the characters doing two jobs — the sentinel values, the silent fallbacks, the "this can never happen" branches, the compat shims nobody remembers writing. Every mature codebase has them. They're invisible precisely because they've never caused a problem anyone could see. Somewhere in your stack right now there's an asterisk quietly meaning two things at once. Go find it before someone with a very long password does.