PyPI Supply Chain Attack: The LiteLLM Lesson

Three files. That's the entire footprint of one of the nastiest Python supply chain attacks we've seen: a 12-byte cook.pth, a 22 KB cook.py, and a 25 KB sitecustomize.py. When LiteLLM 1.107.0 landed on PyPI, those three files rode along inside every one of its seven wheels, and because of how Python's startup machinery works, they executed in every Python process on an infected host — your app, your CI runner, pip itself, even the language server in your editor. You didn't have to import litellm. You didn't have to run it. Starting any Python interpreter was enough.

The LiteLLM team confirmed the compromise and PyPI yanked the release within a few hours, but the window was long enough. The malicious Docker images — including the latest tag — were live for roughly two and a half hours. If your CI pulled during that window, the stealer had already harvested SSH keys, cloud credentials, browser passwords, CI/CD secrets, and AI tool tokens, encrypted them, and shipped them to an attacker-controlled domain. This wasn't a typosquat or a lookalike package. It was the real package, built by the real pipeline, signed by the real release process. And that should change how you think about dependency trust.

How Three Files Hijacked the Official Release Pipeline

The attacker didn't hack PyPI, and they didn't sneak a commit into the public repository — the upstream GitHub source is clean. According to LiteLLM's incident report, the attacker held the GitHub CLI credentials of three developers, apparently harvested from the logs of an internal LiteLLM deployment that was exposed to the internet. With an organization owner's token, they bypassed branch protection and pointed the private release workflow at a lookalike repository, Berri-AI/litellm instead of BerriAI/litellm. A second developer's credential satisfied the approval gate. The official pipeline then built, signed, and published the attacker's artifacts.

The actual injection was elegantly small. One extra line in the sdist's pyproject.toml:

[tool.maturin]
python-source = "."
data = "cook"            # <-- injected, copies cook/ into the wheel
python-packages = ["litellm_cli"]   # legitimate, present upstream

That data = "cook" directive tells maturin to bundle a cook/ directory into the wheel's purelib path, and pip dutifully installs it into site-packages. The sdist itself omitted the folder, so a human skimming the release would see a single cryptic line and nothing else. The payload hid in the build artifacts, not the source. Reviewing the git history — the thing most of us treat as ground truth — would have shown you nothing at all.

Why .pth Files and sitecustomize.py Are a Malware Dream

Two features of Python's startup sequence made this attack unusually potent, and honestly, they're features most Python developers have never thought about. First: any .pth file in site-packages is processed at interpreter startup, and lines beginning with import are executed. The 12-byte cook.pth contained exactly one line — import cook — and that was enough to run the stealer in every Python process on the machine. Second: Python automatically imports a module named sitecustomize at startup if it finds one on the path. The attackers dropped their own, which forked a detached daemon that phoned home to a GitHub-hosted text file every hour, downloaded whatever code the attacker pointed it at, and ran it with exec().

This is a bigger deal than a typical malicious package with a poisoned setup.py or a bad import in __init__.py. Those execute when you install or import the package. This one executed when you ran anything in Python. Your pytest run in CI. The pip list you typed to check what happened. The black formatter in your IDE. Every single invocation re-armed the trap.

It's worth pausing on what the stealer actually collected, because the list reads like a map of modern developer life: AWS, Azure, and gcloud credentials, kubeconfigs, GitHub CLI tokens, SSH and GPG keys, browser password stores, shell histories, CI secrets from GitLab runners and Jenkins, Terraform credentials, environment variables of other processes read via /proc/<pid>/environ, AWS instance metadata credentials, and — with grim irony — tokens for Claude Code, Cursor, Copilot, and a dozen other AI coding tools, plus the LLM provider keys stored in LiteLLM's own proxy database. One pip install, and an attacker holds effectively your entire identity as an engineer.

You didn't have to import litellm. Starting any Python interpreter on the host was enough to run the stealer.

Hash Pinning and Lockfiles: Necessary, Not Sufficient

Every supply chain security checklist starts with lockfiles and hash pinning, and it should. If you pin hashes with pip install --require-hashes or use a tool like uv or Poetry with locked hashes, a wheel that silently changes after you've vetted it gets rejected. That's real protection against a whole class of attacks — registry tampering, mirror poisoning, artifact replacement.

But let's be honest about what it does here: nothing, at least not for whoever upgraded first. Hash pinning guarantees that the artifact you install tomorrow is byte-identical to the one you installed today. In this attack, the malicious wheels were the official artifacts — first-party, pipeline-built, consistently hashed. The moment someone ran their routine upgrade, resolved the new version, and committed the updated lockfile, the malicious hashes were pinned into the repo with a green CI check. Pinning protects you from change; it says nothing about intent.

Where pinning does shine in a story like this is blast radius. Teams with pinned hashes and deliberate upgrade cadences don't automatically ride every release the hour it ships. If your policy is "upgrade dependencies on a weekly PR, with a cooling-off period of a few days for anything that isn't a critical fix," the PyPI removal at 13:23 UTC happened before your upgrade window ever opened. Boring process beats clever tooling more often than we'd like to admit.

So keep your lockfiles. Generate them with hashes. But stop telling yourself they're a defense against a compromised maintainer pipeline — they're a defense against drift and tampering, and a speed bump against bad releases. That's it.

Provenance Attestation Would Have Caught This One

Here's the uncomfortable flip side: this particular attack had a glaring, detectable anomaly, and the ecosystem tooling to catch it exists right now. The malicious wheels were not built from the public repository. They were built from a lookalike repo the attacker created. Any verification step that asks "was this artifact built from the commit and repository it claims?" would have failed loudly.

That's precisely what provenance attestation gives you. PyPI supports PEP 740 attestations: when a package is published via Trusted Publishers from GitHub Actions, the workflow can generate a signed statement binding the artifact to the exact repository, workflow, and commit that produced it. A consumer-side check then becomes mechanical — verify the attestation, confirm the source repo is BerriAI/litellm, not Berri-AI/litellm. Sigstore bundles, SLSA provenance, and pip-audit-style verification all live in this family.

The gap today is adoption and enforcement. Attestations are opt-in for publishers, and almost nobody verifies them at install time even when they exist. That's the part that needs to change, and it rhymes with a point we've made before about trust infrastructure only working when someone actually checks it: a signature nobody verifies is decoration. If your organization runs its own package index or proxy — and if you have more than a handful of engineers, you should — "reject artifacts without verifiable provenance from the expected source" is a policy you can enforce centrally, once, for every developer.

There's a subtler lesson too. The attacker needed three developer credentials: one to redirect the workflow, one to approve, one to spare. That's a systemic failure of credential hygiene, not a single mistake. Short-lived, OIDC-based Trusted Publisher tokens on PyPI would have shrunk the attack surface dramatically — there's no long-lived token sitting in a log file to steal. If your project still publishes with API tokens stored as CI secrets, this is your nudge.

Sandboxed Installs and the Case for Cooling-Off Periods

Assume the worst case: a malicious release slips past your lockfile hygiene and your provenance checks, because someday one will. What's left? Containment. The question stops being "how do we prevent bad packages" and becomes "when a bad package runs, what can it reach?"

For CI, the answer is genuinely achievable. Builds should run in ephemeral environments with no long-lived credentials, scoped tokens, and egress filtering. The LiteLLM stealer's behavior is loud on the network if you're listening: a POST to an unfamiliar domain with a custom auth header, a fetch from a raw GitHub URL on an hourly loop, and — the biggest tell of all — a Python process querying 169.254.169.254, the AWS instance metadata service. Application code almost never has a legitimate reason to hit IMDS directly; the SDK handles it. Alert on Python processes touching that endpoint and you'll catch a whole family of credential thieves, not just this one. We've written before about the blind spots in cloud authentication logging — this is one of them, and it's cheap to close.

Developer machines are harder, and I won't pretend otherwise. A laptop where the same user runs their editor, their browser, their SSH agent, and arbitrary pip installs is a stealer's buffet. The pragmatic moves: do exploratory installs in containers or devcontainers rather than your host environment; keep your package installs inside the project devcontainer so a malicious sitecustomize.py never lands in your host's site-packages; and treat ~/.aws/credentials as a liability — SSO with short-lived tokens (aws sso login, gcloud auth) means there's far less sitting on disk to steal. None of this is exotic. It's just friction, and friction is the point.

• Pin versions and hashes in a lockfile, but treat upgrades as deliberate events, not background noise.
• Impose a cooling-off window — 48 to 72 hours — before non-critical dependency upgrades reach production builds. Most malicious releases are pulled within hours.
• Verify provenance attestations where they exist, and prefer packages published via Trusted Publishers with OIDC over long-lived API tokens.
• Run CI builds in ephemeral, credential-free environments with egress filtering; alert on any Python process touching instance metadata endpoints.
• Install and run third-party code in containers on developer machines, and replace static cloud credential files with short-lived SSO sessions.
• Audit your environment for the IoCs if you pulled anything in the window: cook.pth, cook.py, or a stray sitecustomize.py in site-packages, and /tmp/.pc_once_* or /tmp/remote_code_loader.lock on disk.

What This Attack Says About Where the Real Risk Lives

Notice what the attacker didn't bother with. They didn't exploit a code vulnerability. They didn't need a zero-day. They went for the softest target in modern software: the humans and machines holding standing credentials. Three GitHub CLI tokens, apparently lifted from the logs of an internet-exposed internal deployment, were enough to take over a release pipeline serving millions of downloads. This is the same pattern we've watched repeat across the industry — secrets sprawling into logs, agents, and config files, each one a skeleton key. If you run an internal instance of anything and expose it to the internet, its logs are now part of your threat model. That sentence should make a few people uncomfortable.

There's also a velocity problem worth naming. LiteLLM is a fast-moving project — tens of thousands of commits, frequent releases — and it sits at the center of the AI tooling stack, pulled in by frameworks and SDKs that many teams don't even know depend on it. Transitive reach means you don't get to choose your exposure by choosing your direct dependencies. An SBOM stops being compliance paperwork in this world; it's how you answer "are we affected?" in minutes instead of days. Tools that generate a bill of materials from actual build evidence rather than manifest scraping will catch the dependencies your requirements.txt doesn't mention.

My take: the industry response to supply chain attacks has been overweight on prevention theater — scanners that flag known-bad versions after the fact — and underweight on the unglamorous controls that would have changed this story. Short-lived publishing credentials. Provenance verification at the proxy. Cooling-off periods. Egress alerts on metadata endpoints. Sandboxed installs. None of these are research problems. They're configuration decisions, available today, and most teams have implemented zero of them. The LiteLLM attackers needed one bad line in a build config and three stolen tokens. The defenses above would have required them to beat four independent layers instead of one. That's what defense in depth actually means — not more scanners, but more independent ways to lose. It's the same philosophy behind engineering systems that resist catastrophe rather than ones that merely hope for the best.

The next compromised release is already being planned by someone. You won't stop it from being published. But you can absolutely decide whether it runs with your AWS keys in reach, your SSH agent unlocked, and a clear network path to an attacker-controlled domain — or whether it dies in an ephemeral container with no credentials and a firewall. That decision is made now, in your CI config and your devcontainer setup, not on the day the incident report drops.