A dictionary is the engineering.

The shape of the problem

Blockchain data is a flat stream of bytes that looks, on inspection, like a small alphabet of high-frequency tokens with a long tail of one-off arguments. Take a Solana account snapshot: most of what is in there is the same dozen program IDs, the same Ed25519 key shapes, a small set of common lamport amounts, and the Base58 character set used in addresses. Vanilla Zstandard discovers some of that within a single block of data, but its window is finite and its model resets between calls. It has no notion that 11111111111111111111111111111112 is the System Program.

The shortest path to a better ratio is to tell the encoder up front. Zstandard supports dictionary compression for exactly this case: you build a reference pool of common byte sequences once, and every compress call gets to point at them by offset instead of emitting them.

Why a hardcoded dictionary, not an online learner

Zstandard ships a dictionary trainer. We considered using it. We do not.

A learned dictionary changes between runs, depends on the training corpus, and ships as a separate artifact that the decompressor must locate before it can read anything. That is a reasonable tradeoff for a closed system with one customer, but it is the wrong tradeoff for a library that is supposed to be a single cargo add.

The dictionary in src/presets/solana.rs is a constant. It contains the System Program, Token Program, Associated Token Program, Serum DEX, BPF Loader, Config, Vote, Stake, the four standard instruction discriminators (Transfer, Initialize, Close, Approve), four common lamport amounts (0.01, 0.1, 1, 10 SOL), the signature-count headers used in the transaction structure, and the Base58 character set. Every install gets the same one. Every compressor is deterministic. Round-trip is provable by construction.

The trait layer is generic; the production path is not

The library has three layers, and the boundary between them is on purpose.

src/core/ is generic. It defines CompressionStrategy, the trait every encoder implements, plus a pattern engine that doesn't know anything about blockchains. src/algorithms/ holds three internal algorithms — Enhanced Context Tree Weighting, a multi-pass compressor that iterates until improvement falls below a threshold, and PracticalMaxCompression which orchestrates the previous two. These are the experimental path: they are how we tested whether anything beyond Zstandard moves the ratio meaningfully on real Solana data.

src/presets/solana.rs is the production path. It is just Zstandard plus the dictionary. The reason is honest: when we benchmarked the more aggressive algorithms against vanilla Zstandard-with-dictionary on representative Solana workloads, the marginal ratio improvement did not pay for the throughput cost and the additional surface area. The preset that ships is the one we'd use in production ourselves.

What we give up

Three real tradeoffs. None of them are hidden.

The dictionary is Solana-shaped. Compressing an EVM payload through the Solana preset will still work — Zstandard is a general-purpose encoder — but the dictionary hit rate falls. The ratio numbers in the README do not apply outside the chains the dictionary was built for. If you compress Bitcoin block headers or Ethereum receipt logs, you should benchmark on your own data and expect more modest numbers.

The dictionary is fixed. If Solana introduces a new high-frequency program ID next year, the dictionary will not know about it until we ship a new version. That is the cost of determinism. The alternative — online learning — would have made decompression non-deterministic and made the library awkward to ship.

MaxCompression is slow. Zstd level 19 is not a real-time option. It exists for archival storage, where you write once and read infrequently. If you are compressing on the hot path, use Transactions or FastCompression and look at the lower bound of the ratio range, not the headline number.

The ratio claim, honestly

The README headline says "up to 60:1." The preset table is the load-bearing detail behind that claim: 20–60:1 with MaxCompression (Zstd level 19) on archival workloads. The lower bound, 20:1, is the more useful number for capacity planning. The upper bound, 60:1, is the right number for a benchmark slide if you also publish the workload and the preset. We try not to publish one without the other.

Account state snapshots — by definition full of repeated owner program IDs and serialized account layouts — sit higher in the range. Mixed traffic and one-off transactions sit lower. If you want a number you can trust for your workload, compress a sample of your own data and read the ratio off compressor.stats().best_ratio.

How to engage

The library is MIT-licensed and the source is on GitHub. The preset table is in the README; the design rationale and benchmarks live in the documentation/ mkdocs site. If you find a real workload where the ratio is materially worse than the README range suggests, open an issue with a sample — that is exactly the kind of feedback that earns its place in the dictionary on the next release.