ba64

A binary-to-text encoding that is never larger than base64.

ba64 on GitHub Internet-Draft Spec Security krimler

Here is a thing that happens to every engineer eventually. You have some bytes. Could be a session blob, a saved view, a thumbnail, a protobuf, a signed token. And you have somewhere to put it that only accepts text. A JSON field. A cookie. A log line. A database column that somebody declared as varchar in 2014 and nobody is going to migrate this quarter.

So you base64 it, because that is what everybody does, and it works, and you move on with your life. Then six months later you are staring at a graph of your storage bill, or a cookie that stopped fitting in 4 KB, or a queue rejecting your messages, and you go back and look at that field, and you realise you have been paying a 33% tax on it the entire time.

That is what ba64 is about. It is not a clever new compression algorithm. It is just the observation that if you are going to base64 something, you might as well check whether compressing it first makes the string shorter, and use the shorter one. The whole idea fits in one sentence. The rest of this page is the details that make it safe to actually turn on.

What is base64 actually doing to my data?

Base64 takes three bytes at a time. Three bytes is 24 bits. It chops those 24 bits into four groups of six bits, and writes each 6-bit group as one character from a 64-character alphabet: A-Z, a-z, 0-9, + and /. Six bits has 64 possible values, hence 64 characters, hence the name.

Three bytes in, four characters out. That ratio is where the tax comes from. Four divided by three is 1.333, so a base64 string is always a third bigger than the bytes it came from. If your input length is not a multiple of three, the encoder pads the last group with = characters so the output length stays a multiple of four.

"Hello, world!"        13 bytes
SGVsbG8sIHdvcmxkIQ==   20 characters

Thirteen bytes became twenty characters. Two of those are padding. This is not a flaw in base64, by the way. Base64 was designed in the email era to move binary through channels that would mangle anything outside a safe character set, and it does that job perfectly. It just was not designed to be small, because in 1987 nobody was putting a 30 KB JSON blob in a cookie.

Why does 33% matter enough to bother?

Most of the time it does not. If you base64 a 40-byte key into a config file, who cares. The 33% starts to matter when you hit one of these:

A hard limit you cannot raise. A cookie is 4 KB, and that is the browser's rule, not yours. SQS messages cap at 256 KB. DynamoDB items at 400 KB. Kafka defaults to 1 MB. These limits do not care that your payload is 33% bigger than it needed to be. One day somebody adds three fields to a session object and the cookie stops fitting, and the failure is not a nice exception. It is a cookie that silently does not get set, and users who cannot stay logged in.

Storage you pay for repeatedly. A base64 column is not just bigger on disk. It is bigger in every index, every replica, every nightly backup, and every time it crosses a network boundary. A third more of all of that, forever.

Volume. If you are shipping a billion log events a day and each one carries a base64 field, the 33% is a line item.

And here is the annoying part. The stuff people put in these fields is almost always JSON, or logs, or HTML, or protobuf, or sparse binary. All of that compresses beautifully. You are paying a 33% expansion tax on data that would have shrunk by 70% if anybody had bothered to squeeze it first.

So why not just gzip it before you base64 it?

You should, and lots of people do, and it usually goes wrong in one of three ways.

You forked the format. The moment you write gzip-then-base64 into a field, that field is no longer base64. It looks like base64 to every tool that sees it. It decodes to garbage in every tool that tries. Everything that reads that field now has to know about your convention, including things you do not control and did not think about.

You cannot tell which is which. Old rows have plain base64. New rows have compressed base64. Both are just strings. So you add a prefix like gz:, and now your value is not base64 either, and anything that validates the field with a base64 regex starts rejecting it, and if that field goes in a URL or a cookie you get to find out which characters your prefix broke.

Sometimes it gets bigger. Compression is not free on already-random data. gzip a 4 KB encrypted blob and you get 4 KB plus a header. Now your "optimisation" made the value larger, and unless you check for that every time, you have shipped a size regression that only fires on the payloads you least want to break.

So the requirements write themselves. Whatever you do has to stay inside the base64 character set, has to be self-describing so a decoder knows what it is holding, and must never make anything bigger than plain base64 would have been.

What does ba64 do differently?

It does both. The encoder compresses your bytes with DEFLATE, wraps them in a small header, base64s that, and compares the length against plain base64 of the original. Then it emits whichever string is shorter. If they tie, plain wins.

plain form:       base64(input)
compressed form:  "=" + base64(header + deflate(input))

That is the entire format. Two forms, and a rule for choosing between them.

Here is a real saved-view preferences object, the kind of thing that ends up in a cookie. 239 bytes of JSON:

{"theme":"dark","lang":"en-GB","tz":"Europe/London","density":"compact",
 "columns":["id","customer","status","total","updated"], ... }

base64   320 characters
ba64     237 characters   (74%)

And a batch of twelve log lines, 1,088 bytes:

base64   1,452 characters
ba64       321 characters   (22%)

And here is the same encoder on "Hello, world!" and on a raw UUID:

"Hello, world!"   ->  SGVsbG8sIHdvcmxkIQ==       (identical to base64)
UUID, 16 bytes    ->  jxTkX87qFnpaNt7dS+olQw==   (identical to base64)

Thirteen bytes of English and sixteen bytes of UUID do not compress. There is nothing to find. So ba64 hands back exactly what base64 would have produced, down to the byte. This is the property that makes the whole thing safe to enable: you cannot regress. The worst case is the status quo.

How does a decoder tell the two forms apart?

This is the one genuinely cute part of the design, so let me spend a paragraph on it.

Compressed output starts with =. That character is inside the base64 alphabet, which means every channel that already carries base64 safely will carry it too. Cookies, JSON strings, log lines, HTTP headers: none of them need to change, because = was always going to show up in base64 anyway, as padding.

But padding only ever appears at the end of a base64 string. It is mathematically impossible for a valid base64 string to begin with =, because the first character always encodes real data. So = is a character that is simultaneously safe everywhere base64 is safe, and impossible in any legitimate base64 string.

Which means the decoder is one branch:

if text starts with "="  ->  it is a compressed frame, unwrap it
otherwise                ->  it is plain base64, decode it normally

No prefix that breaks URL validation. No version negotiation. No side channel to tell the reader what it is holding. The string says what it is, in one character, using a character the ecosystem already tolerates.

What is inside the compressed frame?

Four fields and then the payload:

version      1 byte     always 0x01
method       1 byte     0x01 = raw DEFLATE, 0x02 = raw DEFLATE with padding
decoded_len  1-9 bytes  the original length, as a varint
crc32        4 bytes    CRC-32 of your original bytes
payload      rest       a raw DEFLATE stream

The length is there so a decoder can refuse to allocate before it has agreed to the size. A 100-byte string that claims it decompresses to 8 GB is rejected in constant time, without touching the payload. Decompression bombs are a real attack against anything that inflates untrusted input, and the length field plus a caller-supplied cap is what makes them boring.

The CRC-32 is there so a decoder can promise something useful: a damaged ba64 string is always a clear error, never wrong bytes. Corruption in a base64 string usually decodes to plausible-looking garbage, and you find out downstream when something else breaks. Here you find out immediately, with an error code that says which check failed.

To be clear about what the CRC is not: it is not authentication. An attacker who can rewrite your string can recompute the checksum. If you need to know that nobody tampered with the value, you need a MAC or a signature, over the decoded bytes, at the application layer. The CRC catches accidents and bugs, which is most of what actually happens to data in transit.

How much smaller is it, really?

Here is the honest table. It comes from bench/bench.py in the repo, and you can run it yourself in about a second.

A word about those numbers, because benchmark tables lie by default. The first version of this benchmark repeated one log line 120 times and reported 3%. That number was true and completely useless, because no real log file is one line repeated. Every sample here is varied: different IPs, different request IDs, different timestamps, real prose in the HTML. A quarter to a third is what you should expect on text. If you see somebody advertising 5%, look at their input.

Does it still help if my transport already gzips?

This is the first question anybody with an HTTP background asks, and it is the right question. If your response body is already going out with Content-Encoding: gzip, that gzip is running over your base64 string anyway, so what is left for ba64 to do?

More than I expected, which is why that last column is in the table. Compare gzip over base64 against gzip over ba64 and you still save around 30% on text.

The reason is that base64 is actively hostile to gzip. Compression works on repeated byte patterns, and base64 takes your nicely repetitive data and smears it across 6-bit symbols, so the same three input bytes encode to different characters depending on their alignment. The repetition is still in there somewhere, but gzip has a much harder time seeing it. Compress first, while the patterns still line up on byte boundaries, and you keep most of the win even after gzip has had its turn.

Where the transport already compresses and your payload is sparse binary, it is a wash. Where nothing compresses, nothing compresses.

When is it useless?

Often. I would rather you know this now than discover it after a migration.

• Random or encrypted bytes. Keys, IVs, nonces, ciphertext, signatures. There is no structure to exploit, by design. Encryption that produced compressible output would be broken encryption.
• Already-compressed media. JPEG, PNG, MP4, gzip blobs, anything that has already had the redundancy squeezed out of it.
• Hashes. A SHA-256 digest is 32 bytes that look random. It will never shrink.
• Anything small. The frame costs 10 to 14 characters after base64 expansion, so a value under roughly 60 bytes cannot win no matter how compressible it is.

In every one of those cases you get plain base64 back, byte for byte, and you paid one compression attempt in CPU to find out. If your channel carries nothing but keys and UUIDs, ba64 gives you a checksum and nothing else, and you should not bother.

It also does not replace transport compression. If you control the whole body, gzip the whole body. ba64 is for when the individual text field is the unit under pressure, which is exactly the case where transport compression does not help you: the cookie, the queue message, the database column, the log line at rest.

What will it break?

One thing, and it breaks silently, so read this bit.

Do not compare encoded strings. The same input can produce different valid ba64 strings, because DEFLATE output depends on the compression level and the library version. Two services can encode identical bytes and get different text. Both are correct.

So anything that treats the encoded string as an identity breaks: equality checks, deduplication, cache keys, ETags, and worst of all HMAC comparison. Compare decoded bytes instead. Sign decoded bytes instead.

Plenty of systems compare base64 strings today and get away with it, because base64 is canonical: same bytes in, same string out, always. Those are exactly the systems this will break. If you have one, fix the comparison before you turn on the encoder, not after.

Can compressing my data leak secrets?

Yes, in a specific situation, and it is worth understanding because it applies to any compression anywhere near a secret, not just this one.

Compression makes output length depend on content. If an attacker can inject their own text into the same blob as your secret, and can observe the size of the result, then they can guess. A guess that matches the secret compresses slightly better, so the output gets slightly shorter, and they learn that they guessed right. Repeat, one byte at a time. This is CRIME, and its HTTP variant BREACH.

The correct answer is to not compress there. Plain base64 is a conforming ba64 encoder mode, and its length depends only on the input length, so it leaks nothing about content.

If you need compression in that position anyway, the format has an optional pad. You give the encoder a quantum and it rounds the output length up:

ba64.encode(data, pad=64)            # length is always 1 mod 64
ba64.encode(data, pad=ba64.PAD_MAX)  # always len(base64(data)) - 3

At PAD_MAX, two inputs of the same size always encode to the same length, no matter how differently they compress. The attacker's oracle goes quiet.

It costs exactly what it hides. A fully padded value is three characters shorter than plain base64 and no shorter, so you have traded the entire saving for length uniformity. Use it where a hostile party can see the length, and not elsewhere. And it raises the cost of the attack without removing it: enough queries still move a bucket boundary, and one bit always leaks, namely whether the value compressed at all. The full write-up is in SECURITY.md.

How do I roll it out without breaking production?

Compatibility runs one way. A ba64 decoder reads every base64 string ever written, because the plain form is base64. A plain base64 decoder cannot read the compressed form. That asymmetry dictates the order:

1. Readers first. Deploy ba64 decoders everywhere the value can be read. Same call sites, drop-in, no behavior change, because everything in your system right now is plain base64 and decodes identically.
2. Writers second. Turn on the compressing encoder one channel at a time, once you are certain every reader of that channel decodes ba64.
3. Rollback is free. If something is wrong, switch the encoder back to always-plain. That is a fully conforming ba64 encoder. Nothing already written breaks, because your decoders still read both forms.

The hard part is not step 1 or step 2. It is knowing who your readers are. Values escape. They get copied into a spreadsheet, grepped out of a log by a support engineer, read by a partner service that integrated three years ago, or parsed by a script somebody wrote once and left running. A =AQE... in front of any of those looks like corruption, and you will hear about it.

So the honest rule is: if you cannot enumerate the readers of a channel, do not turn on the encoder for that channel. Start with the ones that are entirely inside your own system.

What can I use it from?

Six languages, seven builds. Each one is a single file with no dependencies, so you can vendor it by copying one file into your tree.

Packages go to PyPI, npm, crates.io, Maven Central and NuGet with the v1.0.0 release. Until then, copy the file.

How do I actually call it?

encode takes bytes and returns text. decode takes text and returns bytes, or fails with an error carrying a code from a fixed set of nine. decode also takes a size limit, 64 MiB by default, and that limit is per call, not global, because the right cap for a cookie is not the right cap for a file.

import ba64

text = ba64.encode(b"hello world")   # a string
data = ba64.decode(text)             # the original bytes

try:
    ba64.decode(untrusted)
except ba64.Ba64Error as e:
    print(e.code)                    # for example "E_CHECKSUM"

import { encode, decode, Ba64Error } from "ba64";

const text = encode(new TextEncoder().encode("hello world"));
const data = decode(text);

The browser build is the same API with promises, because the only DEFLATE the web platform hands you is CompressionStream, which is stream-based. No bundler, no wasm, no polyfill.

import { encode, decode } from "ba64/browser";

const text = await encode(new TextEncoder().encode("hello world"));
const data = await decode(text);

text := ba64.Encode([]byte("hello world"))   // string
data, err := ba64.Decode(text)               // []byte, error

let text = ba64::encode(b"hello world");     // String
let data = ba64::decode(&text).unwrap();     // Vec<u8>

How do I know the implementations agree?

Because an encoding is only useful if every implementation of it produces exactly what every other implementation expects, and the only way to believe that is to test it across all of them at once.

There is a shared conformance corpus of 4,084 vectors: plain passthrough, valid frames, every error code with the exact code required, encoder invariants, resource-exhaustion frames, and a 4,000-case differential set. It is generated deterministically and every vector is checked against the reference codec before it is committed. An implementation conforms if it reproduces every decode vector byte for byte and raises every error vector's exact code. Not "an error". The exact one.

On top of that, all seven builds encode the same 10,000 seeded inputs, and then every build decodes every other build's output. That is 49 encoder and decoder pairs, and they agree. Bit-flip and truncation sweeps produce zero silent corruptions across all of them. The Python reference has 100% branch coverage and a 95% mutation score, and every bug ever found is frozen into the corpus as a permanent vector, so it can never come back.

make verify    # corpus regenerates deterministically and passes the reference
make cross     # every language passes the shared vectors
make matrix    # 7x7 encoder and decoder agreement over 10k inputs

Where is the specification?

SPEC.md is the working spec: frame layout, the decoding algorithm as an ordered list of checks so every invalid input maps to exactly one error code, the error taxonomy, resource limits, the padding method, golden examples, and a runnable reference implementation in an appendix.

The same format is an individual, Informational Internet-Draft: draft-gaikwad-ba64. Individual on purpose. Working-group adoption is where somebody proposes negotiation, and somebody else proposes a dictionary registry, and a year later you have a format with four compression methods and a capability handshake. This one has two forms and one rule, and it should stay that way.

Version 1 freezes once the vectors are tagged. There is no version 2 planned. The version byte is an escape hatch, not a roadmap.

Should I use it?

If your text field carries compressible data and you are near a size limit, yes, and the migration is about as safe as format migrations get, because the fallback is the thing you already have.

If your field carries keys, hashes, tokens or images, no. You will get plain base64 with extra steps.

If you are not near any limit and nobody is complaining about storage, also probably no. A third off a field nobody looks at is not worth a coordination problem across every reader you own.

That is the whole pitch. It is a small format that does one thing, it cannot make anything worse, and it is tested harder than it strictly needs to be.