Measuring compression: ratios that matter for archive node operators
Every compression library has a headline ratio. Ours says “up to 60:1” right in the tagline. The preset table backs it up — the MaxCompression preset reports a 20–60:1 range — but the headline number does most of the work in catching attention, and very little of the work in helping anyone make a decision.
This post is for the people making the decision. It is about how to measure compression ratios honestly, what the README’s published ranges actually mean for capacity planning, and what to do when your workload sits at the lower bound instead of the headline.
The shape of an honest ratio claim
A ratio claim is honest if it answers three questions: what input, what preset, what level. “Up to 60:1” answers none of them. “20–60:1 on Solana archival data with MaxCompression, Zstd level 19” answers all three.
Our preset table tries to do this consistently. Each row gives the preset, the underlying Zstd level, and a published ratio range. The ranges are not min/max bounds; they are the band we observed across representative samples of the kind of data that preset is tuned for. If your input is from outside that band — different chain, different shape, mostly random bytes — you should expect to land outside the published range, and probably below the lower bound.
The lower bound is the number to budget against. The upper bound is the number to test against. We will come back to that.
Why the lower bound is the useful number
Picture a procurement conversation. You are explaining to a sceptical finance team why you are confident you can reduce storage costs by a factor of N. The honest answer to “what factor” is: “the lower bound of the preset’s published range, possibly minus a bit for safety.”
If you commit to the upper bound and your actual ratio comes in at the lower bound, you have a problem. If you commit to the lower bound and your actual ratio comes in at the upper bound, you have an unexpectedly easy quarter.
For Accounts, the published range is 15–40:1. Budget against 15:1. Test against 40:1.
For Transactions, the published range is 10–30:1. Budget against 10:1. Test against 30:1.
For MaxCompression, the published range is 20–60:1. Budget against 20:1. Test against 60:1.
Notice that the budget number is still excellent. 10:1 is a 90% reduction in stored bytes. 15:1 is a 93% reduction. 20:1 is a 95% reduction. The headline number is more dramatic, but the budget number is what reduces your storage bill.
How to benchmark on your own data
The crate exposes everything you need to run this measurement on your own corpus. The pseudocode is short:
use blockchain_compression::presets::solana::{SolanaCompressor, SolanaPreset};
use blockchain_compression::core::traits::CompressionStrategy;
let sample = read_sample_from_my_storage(); // pick something representative
let mut compressor = SolanaCompressor::new(SolanaPreset::Accounts);
let original = sample.len();
let compressed = compressor.compress(&sample)?;
let ratio = original as f64 / compressed.len() as f64;
println!("{} -> {} bytes ({}:1)", original, compressed.len(), ratio);
println!("best ratio so far: {}", compressor.stats().best_ratio);Three notes on doing this well.
Pick a representative sample, not a best case. If you measure on a single account snapshot from your highest-traffic program, you will get a number near the upper bound. That is not the number to plan against. Sample uniformly across your data — a hundred randomly-chosen blobs is enough to start.
Measure decompression too. Compression speed often matters less than decompression speed, especially for archival data that is written once and read many times. Time both. The README does not publish speed numbers because they vary too much with hardware, but compressor.stats() tracks both compression_time_ns and decompression_time_ns once you run a real workload.
Round-trip every sample. The crate’s guarantee is that decompress(compress(x)) == x. The right way to verify this on your data is to actually verify it on your data. A single failing round-trip is a bug we want to know about.
Tradeoffs across presets
The published ranges are not the only data you need. Each preset trades ratio against speed differently.
FastCompression (Zstd 3, 5–15:1) is the throughput option. Use it when you are compressing on the hot path — for example, during ingestion, where the alternative is back-pressure. The ratio is lower because Zstd 3 does a much shallower match search, but you can run it inline at multi-gigabyte-per-second throughput on a modern CPU.
Transactions (Zstd 3, 10–30:1) shares the same Zstd level but reports a meaningfully better ratio range. That difference is the dictionary at work. Transactions are exactly the data the Solana dictionary was tuned for; it fires often, the back-references are short, and the resulting bytes-per-token cost falls.
Instructions (Zstd 6, 10–25:1) and Accounts (Zstd 6, 15–40:1) both step up to Zstd level 6. The match search is deeper, the encoder spends more time looking for slightly-longer matches, and the result is a noticeably better ratio on the kind of data each is tuned for. The CPU cost is higher than Zstd 3 but still well within the budget of most batch pipelines.
Mixed (Zstd 6, 12–35:1) is the right choice when your data is heterogeneous. It is the same Zstd level as Instructions and Accounts with the same dictionary, and the published range is wider because the input is more varied.
MaxCompression (Zstd 19, 20–60:1) is the archival option. Zstd 19 is dramatically slower than Zstd 3 — somewhere between 50× and 100× depending on the input — but it finds matches the lower levels miss. Use it when you are writing to cold storage and you will rarely re-read the data. The asymmetry favours the archival case: Zstd compression is the slow part; Zstd decompression is fast across all levels.
What changes when the ratio sits at the lower bound
Suppose you measure on your own data and the ratio for Accounts comes in at 14:1 — just below the published lower bound of 15:1. What does that tell you?
Most likely, your data is slightly different from the data the dictionary was trained on. The dictionary contains the eight or so program IDs that dominate generic Solana traffic. If your workload is dominated by a different program — your own protocol, a less-common DEX, a custom token standard — the dictionary fires less often, and the ratio drops toward the vanilla-Zstandard baseline.
Two reasonable responses:
The first: try MaxCompression instead. The deeper match search at Zstd 19 will find more cross-block patterns within the larger window, partially compensating for a dictionary that is not perfectly tuned to your specific traffic. If the bill is storage and you do not re-read often, this is usually the right call.
The second: open an issue with a sample. The dictionary lives in src/presets/solana.rs and is shipped as a constant. If the high-frequency byte patterns in your real workload are not in there, the right fix is to add them. The dictionary stays a constant, decompression stays deterministic, and the next release picks up a better ratio for everyone running the same kind of workload.
What changes when the ratio sits at the upper bound
If you measure and the ratio comes in at 50:1 on Accounts — well above the upper bound — something is unusually repetitive about your data. Account snapshots from a single program with identical layouts compress like this. Backups of a hot cache with duplicate entries compress like this. So do pathological samples — a single account replicated a thousand times.
The honest move is to measure on more samples. If 50:1 reproduces across a uniformly-sampled corpus, congratulations: your workload sits at the easy end of the spectrum, and you have a real budget story to take to finance.
The summary, with the numbers
The right way to talk about this library’s compression ratios:
| Preset | Budget | Realistic | Best case |
|---|---|---|---|
| FastCompression | 5:1 | ~10:1 | 15:1 |
| Transactions | 10:1 | ~20:1 | 30:1 |
| Instructions | 10:1 | ~18:1 | 25:1 |
| Accounts | 15:1 | ~27:1 | 40:1 |
| Mixed | 12:1 | ~23:1 | 35:1 |
| MaxCompression | 20:1 | ~40:1 | 60:1 |
The “budget” column is the lower bound of the README’s published range — what we recommend committing to. The “realistic” column is the midpoint, an order-of-magnitude estimate. The “best case” column is the headline number — what we tell people to test against, not what we tell them to plan against.
That distinction is the entire honesty story.