How we fit 16 bits into every LLM token — and reverse-engineered Claude's tokenizer through a free API.
Every LLM you talk to runs on top of a compressor. That's not a metaphor — byte-pair encoding was invented in 1994 as a compression algorithm, and a tokenizer is exactly that: it maps frequent byte sequences to single symbols so the model sees ~4 characters per token instead of one. Karpathy's tokenizer lecture makes the point directly: the vocabulary is a codebook.
Here's the thing about compressors: they have a rated capacity, and they almost never run at it. An o200k token can carry log2(200,000) ≈ 17.6 bits of information. English prose runs the channel at ~5–6 bits per token. Source code: about the same. Base64 — the thing everyone reaches for when they need to park binary data in a prompt — manages 8.8 bits per token. Half the channel, wasted, because BPE merges base64 characters unpredictably across chunk boundaries.
That gap between rated and actual capacity is slack. We built densely to fill it.
Suppose you want to store compressed bytes inside a context window — losslessly, so the agent can get the exact original back later. You need an encoding whose token cost is guaranteed, not average-case.
The trick: pick 65,536 English words that each cost exactly one token when preceded by a space. Map every 16-bit chunk of your lzma stream to one word. The payload comes out looking like a stream of random words — the of carbon whispers… — and every one of those words is exactly one token. Two bytes per token. 16 of the ~17.6 available bits: 91% of channel capacity, versus base64's 50%.
The "guaranteed" part is provable, and this is my favorite detail. o200k's pre-tokenizer regex is public — it ships in tiktoken. It splits text into chunks of the form "optional single non-letter + letters" before byte-pair merges run, and BPE cannot merge across chunk boundaries. So " word word" is always exactly two tokens, for any two alphabet words, in any order. Worst case equals average case. No benchmark required; the regex is the proof.
(The same regex explains our dead ends: digit strings get split into groups of 1–3, CJK merges greedily with neighbors — we probed both and the math said no.)
Then a Reddit commenter found the hole. Our alphabet was calibrated on o200k — OpenAI's tokenizer. On Claude, whose vocabulary is not public, those same 65,536 words averaged 2.98 tokens each. Effective density: 5.4 bits per token. A third of the claim — and nothing looked broken. Reconstruction stayed byte-exact; you just quietly paid triple.
Anthropic doesn't publish their vocab. But they do give you a free count_tokens endpoint. That's an oracle: you can ask "how many tokens is this string?" as many times as you like.
So we harvested a Claude-native alphabet empirically:
1 — Take ~68k candidate words, frequency-ordered so the useful ones come first.
2 — Batch 32 words into one string; if the count comes back 32, every word is single-token — keep the whole batch.
3 — If not, bisect the batch to isolate the multi-token offenders.
4 — Calibrate away the request overhead first, and don't use the "obvious" two-request subtraction: c(A)+c(B)−c(A+B) is off by one, because tokens retokenize across the junction. Count "the" repeated N times instead — that's exactly N tokens plus overhead.
~45,000 probes later: Claude's tokenizer contains roughly 1,035 single-token space-prefixed words in total. Not 68k — one thousand. The word-class carrier space on Claude is simply exhausted at 210. So the Claude alphabet is 1,024 words at a guaranteed 10 bits per token — 1.85x denser than what the o200k alphabet actually delivered there.
Verified end-to-end against Sonnet 5's own counter: a real 172KB log went 93,117 raw tokens → 25,904 payload tokens. The alphabet-math prediction matched the live API within 0.1%.
Same pipeline, pointed at whatever was lying around, counted by the tokenizer APIs that do the billing. Byte-exact round trip verified per row.
| Data | Claude raw → payload | Saved |
|---|---|---|
| macOS /var/log/install.log slice (195KB) | 104,328 → 6,732 | 93.5% |
| Live npm registry JSON (express) | 113,444 → 19,672 | 82.7% |
| A real package-lock.json | 15,743 → 5,910 | 62.5% |
| Source code (this repo, pip internals) | — | 36–40% |
Code saves least — code is already token-dense; the tokenizer was trained on it. Logs save most: they're the most redundant thing agents ever read, and they're exactly what agents read all day.
The standard objection: the model can't read the payload, so what's the point?
Correct — and deliberate. It's cold storage plus targeted retrieval: a server-side regex search returns only matching lines (with line numbers); expand returns exact ranges, sha256-verified. Counting the errors in a 74k-token log costs you the 15 matching lines. The full text never needs to exist in context.
Jeff Dean sketched the endgame in his Princeton colloquium this year: attending to a trillion tokens won't come from bigger windows alone, but from hybrid systems — retrieval over large corpora, lightweight relevance filtering, and only the genuinely needed pieces entering the window. He also gave the reason exactness matters, better than we ever phrased it: information in the context window is valuable because it's crisp — "you haven't mixed it with anything else" — unlike training data, "stirred together" into weights. Lossy summarization destroys precisely that crispness. The stack trace that comes back as "an error occurred" is muddled context wearing crisp context's clothes.
Our bet, stated plainly: for production agents, context exactness becomes a correctness property — like types, like tests. "Approximately the right bytes" will eventually sound as absurd as "approximately the right arithmetic."
Everything in densely follows from that one belief: lzma because it's boring and deterministic, sha256 on every expand, carrier alphabets proven per-tokenizer rather than measured on average.
Honesty section, same as the README. If you control your harness end-to-end, build this yourself in an afternoon: zstd the big outputs to files, give the agent grep. densely is that pattern packaged — it earns its keep inside closed harnesses (Claude Code, Cursor), ephemeral sandboxes, and wherever "reliable disk + grep" doesn't reach. Actively-edited code gains nothing — payloads are unreadable, don't compress what the model must read. Single-shot tasks lose: break-even is 3–4 turns. And Shannon always wins on already-dense data.
pip install "densely[mcp]"
claude mcp add --scope user densely -- "$(which densely-mcp)"MIT, all of it — including the alphabet harvester (tools/build_alphabet.py), so you can scan any tokenizer, and tools/calibrate.py, so you don't have to trust a single number in this post.