Why generic compressors leave bytes on the table for chain data

Vanilla Zstandard is excellent. It is the default we benchmark against, the encoder we reach for outside this library, and the encoder this library uses under the hood. The reason a separate crate exists is not that Zstandard is bad. It is that Zstandard does not know what it is looking at.

This post is the long version of that observation. It walks through what generic compressors actually do on Solana data, where the byte-level pattern repetition is, and how a hardcoded dictionary changes the math.

What a generic LZ-family compressor sees

Zstandard, like every modern general-purpose compressor, is built on the LZ77 family. The mental model is simple: as the encoder reads the input stream, it maintains a sliding window of recently-seen bytes. When it encounters a sequence it has already seen inside that window, it emits a back-reference — an offset and a length — instead of the bytes themselves. Then it Huffman-codes the references and the literals together, and that is most of the magic.

This works extraordinarily well on text. It works extraordinarily well on logs. It works on serialized JSON, on database dumps, on most things you would throw at gzip without thinking. The reason is that those inputs are full of local repetition. The phrase “ERROR” appears twice in two adjacent log lines, and the second appearance becomes a one-byte token in the output.

It works less well on data where the repetition is non-local. The Solana System Program ID, 11111111111111111111111111111112, appears in nearly every transaction — but the previous transaction’s appearance was likely outside the encoder’s sliding window by the time the next transaction is being compressed. So the encoder pays for it again. And again. And again.

Why the window matters

The DEFLATE specification defines a 32 KB sliding window. Zstandard uses a larger configurable window — up to a gigabyte at the most aggressive levels — but in practice the window is still finite, and it is reset between compress calls. If you compress one block at a time, each block starts cold. The first block pays full price for every common byte sequence. The second block starts cold and pays again.

You can paper over this with concatenation: glue blocks together and compress the result as one stream. That works, but it changes the access pattern. You can no longer decompress a single block; you have to decompress the entire stream up to the block you want. For archival storage that is sometimes fine. For an indexer that needs random access to past account state, it is not.

The other lever is to compress more aggressively. Zstd levels 19 and 22 use much bigger windows and run a much more thorough match search. That is the lever the MaxCompression preset pulls. It gets you a long way, but at a real CPU cost — Zstd 19 is between 50× and 100× slower than Zstd 3 — and it still does not know that the byte sequence in front of it is a Solana program ID.

What a hardcoded dictionary changes

Zstandard supports something the DEFLATE spec does not: a custom dictionary, supplied at encoder construction. The dictionary is treated as if it had appeared in the stream just before the actual data. Any reference into the dictionary becomes a tiny back-reference token in the output. The dictionary itself is not stored in the output — both sides of the conversation are expected to know about it.

That is exactly the shape of the problem. We do not need the encoder to discover that TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA is the Token Program. We can tell it. We can also tell it about the System Program, the Associated Token Program, the four standard SPL token instructions, the lamport encodings of 0.01, 0.1, 1, and 10 SOL, and the Base58 character set used in every address encoding on the chain. Each of those byte sequences becomes a back-reference in the output, and each occurrence in the input gets collapsed to a few bytes.

The full list of patterns lives in src/presets/solana.rs under SOLANA_DICTIONARY_PATTERNS. It is short, hand-curated, and tied directly to the Solana programs and conventions that show up at high frequency in real chain data.

What this actually buys you

The preset table in the README puts published ratio ranges next to each preset. Those ranges are the ones we observed on representative Solana workloads. They are not promises. They are not guarantees on arbitrary data. They are the band you should expect if your data looks like the kind of data the dictionary was tuned for.

Two of those numbers are worth pulling out.

For the Accounts preset, the published range is 15–40:1. Account snapshots are unusually amenable to dictionary compression because most of the bytes in a snapshot are repeated owner program IDs, repeated rent-exempt minimum amounts, and the serialized layouts of a small set of common account types. The dictionary catches most of that on the first scan.

For the MaxCompression preset, the published range is 20–60:1. That is the headline number in the tagline, and it is the right number to quote in a benchmark slide as long as you also publish what you were compressing. The 60:1 upper bound is what you get on archival workloads where data is unusually repetitive at the byte level — and where the cost of Zstd level 19 is acceptable because you write once and read rarely.

The lower bound is the more useful number for capacity planning. If you store account state and you bet on 15:1 with the Accounts preset, you will not be disappointed. If you bet on 40:1, you might be.

What vanilla Zstandard still does well

A useful framing: this library is exactly Zstandard, plus a hardcoded dictionary, exposed through a preset table that maps named use cases onto Zstd compression levels. Everything Zstandard does well — fast decompression, deterministic output, well-understood operational behavior, broad platform support — is preserved. The places where the library wins are the places where the dictionary fires.

The places where vanilla Zstandard is still the right answer:

• Non-Solana data. If your input is HTTP responses, log files, or arbitrary blobs, the Solana dictionary is dead weight. It will not hurt your ratio in any meaningful way, but it will not help either. Use plain zstd and a level that fits your CPU budget.
• Heterogeneous data where you cannot identify the shape. The Solana presets assume your input is Solana data. If you are running a multi-chain indexer and need one compression path for everything, vanilla Zstandard with a moderately high level is a fine answer.
• Streaming compression of a long stream. Once your stream is long enough that the cross-block repetition fits inside the sliding window, the dictionary’s contribution shrinks. The places the dictionary wins most are short, independently-decompressable blocks — exactly the access pattern an indexer wants.

How to know if the dictionary fires for you

The crate exposes compressor.stats().best_ratio after a compress call. If you are evaluating the library, the most useful thing you can do is point it at a sample of your actual data with each preset and look at that number. If it is in the README’s published range, the dictionary is working for your workload. If it is materially below the lower bound, your data is probably not what the dictionary was trained for, and you would do better either with a learned dictionary on your specific corpus or with vanilla Zstandard at a higher level.

That measurement takes about ten minutes. We have seen people make capacity-planning decisions on the headline 60:1 number without it. Do not be those people. The library is honest about its ratio range because the alternative — pretending every workload hits the upper bound — would be the kind of marketing claim that survives until the first procurement call.

The takeaway

Generic compressors are excellent at the general case. Chain data is not the general case. It is a stream of bytes drawn from a narrow alphabet of high-frequency tokens — program IDs, instruction discriminators, lamport amounts, Base58 — that a generic encoder has no a priori knowledge of. Hardcoding that knowledge into a Zstandard dictionary is, frankly, not interesting from a compression-theory standpoint. It is just the right engineering. The ratio improvement falls out of being slightly less abstract about the problem than the general-purpose encoder is allowed to be.