OpenSSL EVP_EncodeBlock Overflow: 5 Myths, Tested
OpenSSL's EVP_EncodeBlock buffer overflow (CVSS 9.8) shows how base64 encoder APIs escape scrutiny. Five myths about encoder security, tested.

A function that does nothing but turn bytes into base64 text just earned a CVSS score of 9.8. The EVP_EncodeBlock buffer overflow in OpenSSL — tracked as CVE-2026-121481 — is the kind of bug that sounds boring until you realize base64 encoding sits underneath certificate handling, HTTP clients, proxy authentication, and half the glue code in your infrastructure. The fix itself took four commits and a genuinely heated argument among maintainers about what an error return should even look like.
The discussion around this CVE is a gift for anyone who cares about API design, because it surfaces a pile of assumptions engineers make about encoding functions — assumptions that keep producing the same bug class across decades and across libraries. Let's take five of the most common beliefs and hold them up against what actually happened.
How the EVP_EncodeBlock Buffer Overflow Actually Works
Base64 encoding inflates data: every three input bytes become four output characters. So the output size is 4 * ((n + 2) / 3), plus one for a NUL terminator if you want a C string. OpenSSL's EVP_EncodeBlock() makes the caller compute that size and allocate the destination buffer. The function itself receives the input pointer, the output pointer, and the input length — as an int. It never learns how big the output buffer is, so it cannot validate anything. It just writes.
Now do the arithmetic. The largest int is 2,147,483,647. When the input length exceeds (INT_MAX / 4) * 3 — that's 1,610,612,733 bytes, roughly 1.6 GB — the internal size computation overflows a signed 32-bit integer. The length math wraps, the function's idea of where output ends no longer matches reality, and you get writes past the end of whatever buffer the caller allocated. The caller, following the documented formula, did the same overflowing math, so the two sides can disagree catastrophically about how much room exists.
The upstream fix added a guard: reject inputs where dlen > (INT_MAX / 4) * 3, along with NULL buffers and negative lengths. If you wrap this API — and you should wrap it — the check belongs at your boundary too:
#include <limits.h>
#include <stddef.h>
#include <openssl/evp.h>
/*
* Mirrors the upstream guard: refuse inputs whose encoded size
* cannot be represented in an int, and verify the destination
* capacity ourselves since the API never sees it.
*/
int b64_encode_checked(const unsigned char *in, size_t n,
unsigned char *out, size_t out_cap)
{
size_t need;
if (in == NULL || n > ((size_t)INT_MAX / 4) * 3)
return -1;
need = ((n + 2) / 3) * 4 + 1; /* +1 for the NUL terminator */
if (out == NULL || out_cap < need)
return -1;
if (EVP_EncodeBlock(out, in, (int)n) < 0)
return -1;
return 0;
}
Notice what this API forces you to do: reimplement the library's internal size formula, in the caller, correctly, forever. That contract is the real vulnerability. The overflow is just the place where the contract finally snapped.
Myth 1: "Decoders Are the Attack Surface, Encoders Are Boring"
Security review culture has a strong bias: parsing is dangerous, serializing is bookkeeping. Decoders consume hostile bytes, so they get fuzzed, audited, and rewritten in memory-safe languages. Encoders produce output, and output feels safe. This belief is wrong in a specific, repeatable way: encoder bugs live in the output-size computation, and that math is trusted by both sides of the call.
The pattern keeps recurring. The PR thread itself notes that CVE-2026-121481 resembles CVE-2021-23840, an earlier OpenSSL overflow in the EVP update functions where a length computation overflowed on huge inputs. And base64 specifically has form here — when OpenJDK's Base64 decoder turned 65 bytes into remote code execution, we covered how a "trivial" codec became an RCE primitive. Decoders get CVEs for accepting bad input; encoders get CVEs for miscalculating good input. Both are the same species: arithmetic on attacker-influenced lengths.
Verdict: busted. If your threat model stops at the parser, you've left the back door open while guarding the front.
Myth 2: "CVSS 9.8 Means Drop Everything Tonight"
Here's where I'll defend the skeptics a little. A 9.8 on an encoder that needs a 1.6 GB input is not Heartbleed. There is no base64 encoding of bulk data anywhere in the TLS handshake — certificates travel as DER inside TLS, not PEM. For this to burn you, untrusted data has to reach EVP_EncodeBlock() (or an internal caller like base64encode() in OpenSSL's HTTP client) at a scale above 1.6 GB. A protocol field with a 16-bit length prefix can never get there.
But "synthetic" undersells it too. The reporter demonstrated a real command-line path: pass an enormous -proxy_pass value to openssl s_client, supplied through an environment variable to dodge shell argument limits, and certain configurations — GOST engines accept oversized passwords out of the box — will happily base64-encode it for a proxy CONNECT request. That's not a lab curiosity; that's a standard tool doing standard proxy auth with a standard environment variable.
My position: treat CVSS as a measure of the function, and your call graph as the measure of your exposure. Before you schedule an emergency patch window, grep your codebase and your dependencies for who actually encodes what, and how big the input can get. Then patch anyway — just in the right order. Verdict: half true. The score is real, but the panic should be proportional to reachability, and reachability is a question only your code can answer.
Myth 3: "The Fix Is Easy — Just Return an Error Code"
This is my favorite part of the whole story, because the overflow was the simple bit. The hard bit was deciding what EVP_EncodeBlock() should do about it — and the maintainers could not agree.
The original patch returned -1 for invalid inputs, matching wolfSSL's compatibility layer and the existing failure paths in EVP_DecodeBlock(). Pushback came immediately. One maintainer worried that third-party callers cast the return value to size_t without checking — and -1 becomes SIZE_MAX under that cast, which is arguably worse than the overflow it prevents. So return 0 instead? Except 0 is already a legitimate success: encoding zero bytes returns zero. As one maintainer put it, he'd already been burned for raising an error on a success case. Use OPENSSL_assert() then, said another — crash on contract violation and document the precondition. To which a senior maintainer replied, flatly: an assert inside a library is "a DoS waiting to happen."
Every option was bad, and each was bad for a different reason: -1 breaks unsigned casts, 0 collides with success, assert turns a logic error into an attacker-controlled crash. This is what happens when an API's error semantics were never designed, only inherited. It's the specification-implementation gap in its purest form — the documentation said "returns the number of bytes encoded," and nobody had ever written down what happens when encoding is impossible.
Two more details from the thread deserve attention. First, a reviewer caught that the initial patch rejected f == NULL unconditionally — but encoding an empty buffer with a NULL input pointer and dlen == 0 had always worked, and real applications depended on it. Even the "obviously correct" validation broke a legitimate edge case. Second, a reviewer grepped for call sites that don't check the return value and found them inside OpenSSL itself: PEM_SignFinal() and NETSCAPE_SPKI_b64_encode(). Making a function newly fallible does nothing if its callers were written when it couldn't fail. Verdict: busted, thoroughly. Error semantics are API design, and retrofitting them is surgery, not a patch.

Myth 4: "Nobody Feeds a Gigabyte Into a Base64 Encoder"
The intuition here is that base64 is for small things: certificates, keys, short tokens. And mostly that's true. But modern systems have quietly grown enormous blobs that get encoded all the time — attestation documents, SAML responses, container image manifests, proxy credentials, webhook payloads that some middlebox decides to re-encode. The reproducer for this CVE wasn't exotic: it was a password, delivered through an environment variable, on a code path that encodes credentials for an HTTP proxy. Nothing about that pipeline was designed with a 1.6 GB ceiling in mind, because nobody wrote the ceiling down anywhere.
There's also an architectural habit that makes this worse: one-shot encoding APIs encourage reading an entire payload into memory before transforming it. OpenSSL has streaming alternatives — EVP_EncodeInit(), EVP_EncodeUpdate(), EVP_EncodeFinal() — that process data in chunks and sidestep the whole "compute the total output size up front" problem. Streaming costs you a context object and a loop. What it buys is that no single length ever has to be both attacker-influenced and trusted. Verdict: busted. Inputs grow until they hit a limit, and if the limit is a signed integer overflow, the limit finds you.
Myth 5: "Our Fuzzers Would Have Caught This"
They wouldn't, and it's worth understanding why. Coverage-guided fuzzers like libFuzzer default to maximum input lengths measured in kilobytes — 4096 bytes out of the box. A bug that requires 1.6 billion bytes of input is not merely unlikely to be found; it is outside the search space entirely. You could fuzz EVP_EncodeBlock() for a century and never brush against the overflow, because the interesting behavior only exists at a scale the fuzzer will never visit.
The fix isn't bigger fuzz inputs — nobody wants to allocate 1.6 GB per fuzz iteration. The fix is to test the arithmetic separately from the encoding. Extract the size computation and hammer its boundary with plain unit tests:
#include <assert.h>
#include <limits.h>
/* Mirrors the upstream guard, computed in a wide type. */
static int encoded_len_fits_int(long long dlen)
{
return dlen >= 0 && dlen <= ((long long)INT_MAX / 4) * 3;
}
int main(void)
{
long long limit = ((long long)INT_MAX / 4) * 3; /* 1,610,612,733 */
assert(encoded_len_fits_int(0));
assert(encoded_len_fits_int(limit));
assert(!encoded_len_fits_int(limit + 1));
assert(!encoded_len_fits_int(-1));
/* The encoded size at the limit must itself fit in an int. */
assert(4 * ((limit + 2) / 3) <= INT_MAX);
return 0;
}
Then add a differential check at moderate sizes: compute the expected output length with your caller-side formula, encode, and assert the function wrote exactly that many bytes and no more. Run everything under AddressSanitizer with a few hundred-megabyte inputs in CI — large enough to exercise the wide loops, small enough to finish before lunch. Fuzzing is still the right tool for the codec's byte-level logic; it was just never the right tool for integer-overflow-at-scale, and confusing those two jobs is how bugs like this survive a decade of continuous fuzzing. Verdict: busted.
How to Audit an Encoder API Before It Audits You
Whether you're reviewing OpenSSL usage in your own code or evaluating any library's encode path, the same handful of questions catches this entire bug class:
- Who computes the output size? If the caller does it with a nontrivial formula, both sides can drift. Prefer APIs that take the output capacity as a parameter and fail cleanly when it's insufficient.
- What integer type carries lengths? An
intlength parameter is a 2 GB ceiling with a sign bit attached.size_tdoesn't make overflow impossible, but it moves the ceiling past the edge of any buffer you can actually allocate on a 64-bit machine. - What happens at zero, at the boundary, and one past it? Empty input, exactly
(INT_MAX/4)*3, and that value plus one are the three test cases that matter. - Did the function recently become fallible? If a patch added error returns to something that couldn't fail before, grep every call site. Unchecked returns turn a fixed bug into a silent one.
- Is there a streaming variant? Chunked encode/update/final APIs remove the need for any single trusted total length.
I'll go further on the first point: any API that makes the caller replicate the library's internal size formula is a bug factory, and the industry should stop building them. The callee knows the expansion factor, the padding rules, and the terminator convention. Forcing every caller to re-derive 4 * ((n + 2) / 3) + 1 guarantees that someone, somewhere, gets it wrong — or gets it right in a type that's too small.

The Takeaway for Base64 and Beyond
CVE-2026-121481 will fade from the feeds within a week, but the bug class won't. Encoding functions sit in the blind spot of security review: trusted with lengths, excused from fuzzing at scale, and wrapped in APIs whose error behavior was never specified. The OpenSSL maintainers spent more words arguing about whether to return -1, 0, or crash than they did on the overflow itself — and they were right to, because that's the part every downstream caller will live with for years.
So: patch, yes, but also grep. Find your encode call sites, check who computes the buffer size, verify the returns are actually checked, and move anything that handles untrusted bulk data to a streaming API. The next 9.8 in an encoder is already sitting in someone's size formula, waiting for an input big enough to find it.


