crate · blockchain-compression · v0.1 · MIT
State bloat is a storage problem.
Compress it up to 60:1 on archival Solana data.
Every high-throughput chain grows faster than storage gets cheap. blockchain-compression pairs
Zstandard with a custom dictionary of Solana byte patterns — program IDs, instruction
discriminators, lamport amounts, Base58 — so archive nodes, indexers, and DePIN storage
networks keep and move less data. The one measured headline is
up to 60:1 on archival workloads with
MaxCompression. Other preset ranges are illustrative and depend
on how repetitive your input is: the README reports 20–60:1
with MaxCompression, 15–40:1
for account snapshots, and 10–30:1 for transactions.
// ratio range by preset (from README)
lower = better compressed size
// the problem
State grows forever. Storage budgets don't.
A high-throughput chain adds state every slot and never gives it back. Full history becomes terabytes, then tens of terabytes, and the people who keep that history online — archive-node operators, indexers, explorers, and the DePIN storage networks paid to hold it — carry the bill. Generic compressors help, but they don't know what a chain looks like: a vanilla zstd encoder discovers some structure within a single block, never learns anything across blocks, and has no opinion about the same program IDs, signature shapes, and lamport amounts that repeat billions of times. Closing that gap is cheaper archival and cheaper data movement — which, as chains scale, is core infrastructure. That is the entire reason this library exists.
// archival
Cold storage scales with the chain
Account-state snapshots accumulate for the life of the network. Once you are holding terabytes of write-once history, a high archival ratio is the difference between a flat storage line and a cost curve that tracks block height.
// data movement
DePIN networks pay to move bytes
Decentralized storage and RPC networks are billed on what they hold and what they replicate across peers. Fewer bytes on the wire means cheaper replication and less bandwidth burned shipping the same repeated program IDs to every node.
// serving history
Explorers and indexers query the tail
Indexers warehouse the firehose of decoded transactions; explorers serve the long tail of history. Smaller stored size keeps more of the chain on warm disk, so fewer queries pay the cold-tier round-trip.
// what's in the dictionary
Hardcoded Solana patterns, not online training.
The dictionary is built once at SolanaCompressor::new()
from a fixed list of byte patterns. That list lives in
src/presets/solana.rs.
No per-customer state, no online learner, no model file shipped beside the binary.
The same dictionary on every install — deterministic round-trip by construction.
Zstandard receives that dictionary at encoder construction time and uses it as a reference pool. Byte sequences that match the dictionary become tiny back-references in the output stream.
// SOLANA_DICTIONARY_PATTERNS
// src/presets/solana.rs (excerpt)
const SOLANA_DICTIONARY_PATTERNS: &[&str] = &[
"11111111111111111111111111111112", // System Program
"TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA", // Token Program
"ATokenGPvbdGVxr1b2hvZbsiqW5xWH25efTNsLJA8knL", // Associated Token
"Vote111111111111111111111111111111111111111", // Vote Program
"Stake11111111111111111111111111111111111111", // Stake Program
"00e1f50500000000", // 1 SOL in lamports
"00ca9a3b00000000", // 0.1 SOL in lamports
// ... transfer / initialize / close / approve discriminators
];// how it works
A chain-agnostic core, a Solana dictionary on top.
The crate layers a generic pattern engine under a Solana-specific preset. Bytes flow through a compression strategy that references the dictionary, then Zstandard emits the stream. Nothing in the pipeline knows about a running validator.
// core/
Chain-agnostic engine
The CompressionStrategy trait and a generic
pattern engine. No chain knowledge lives here — it is reusable machinery.
// algorithms/
Aggressive passes
EnhancedCTW,
MultiPassCompressor, and
PracticalMaxCompression back the higher presets.
// presets/solana.rs
The production path
SolanaCompressor builds the Zstd dictionary
at init and passes it to the encoder. This is what you use in practice.
// presets
Six presets. Each one a Zstd level plus the same dictionary.
Ratios below are the README's published ranges. Real numbers depend on how repetitive your input is — a single random EVM payload won't compress like a batched Solana account dump.
| Preset | Zstd level | Reported ratio | Best for |
|---|---|---|---|
| FastCompression | 3 | 5–15:1 | Real-time processing |
| Transactions | 3 | 10–30:1 | Transaction data |
| Instructions | 6 | 10–25:1 | Program instructions |
| Accounts | 6 | 15–40:1 | Account state snapshots |
| Mixed | 6 | 12–35:1 | General-purpose |
| MaxCompression | 19 | 20–60:1 | Archival storage |
// guarantees
Lossless or it's a failing test.
The README is explicit: "100% lossless — rigorous roundtrip validation on every
operation." Every preset's test suite compresses sample data and asserts that
decompress(compress(x)) == x. That guarantee
is the contract of the CompressionStrategy
trait, and it is the same contract whether you reach for
FastCompression or
MaxCompression.
- ✓Round-trip checked in unit tests for every preset.
- ✓Deterministic dictionary: same bytes in, same bytes out, anywhere.
- ✓Thread-safe — safe to share across concurrent encoders.
- ✓No network, no telemetry, no chain dependency at runtime.
// roundtrip.rs
use blockchain_compression::presets::solana::{
SolanaCompressor, SolanaPreset,
};
use blockchain_compression::core::traits::CompressionStrategy;
let mut c = SolanaCompressor::new(SolanaPreset::Transactions);
let compressed = c.compress(data)?;
let decompressed = c.decompress(&compressed)?;
assert_eq!(data, decompressed.as_slice());
// If this fails, the library is failing its own
// test suite — open an issue.// install
Drop it into Cargo.toml.
// Cargo.toml — Zstd backend (recommended)
[dependencies]
blockchain-compression = { version = "0.1", features = ["zstd"] }
The Zstd backend is required for the Solana presets. DEFLATE and LZ4 are available as
alternative backends behind features = ["deflate"]
or features = ["lz4"].
// main.rs — basic usage
use blockchain_compression::presets::solana::{
SolanaCompressor, SolanaPreset,
};
use blockchain_compression::core::traits::CompressionStrategy;
let mut compressor = SolanaCompressor::new(
SolanaPreset::Accounts,
);
let compressed = compressor.compress(data)?;
let decompressed = compressor.decompress(&compressed)?;
assert_eq!(data, decompressed.as_slice());// honest comparisons
Where this library fits.
The reference points worth comparing against are general-purpose compressors that almost every node operator already has installed.
vs vanilla zstd
→Same encoder underneath. The difference is the custom dictionary of Solana byte patterns — references collapse to small tokens that zstd alone would have to discover within a single block.
vs DEFLATE / gzip
→DEFLATE is everywhere and fast enough for most things. But it has no dictionary support and no facility for cross-block context, which is where the Solana preset earns its ratio.
// explore
Everything, one link away.
Every page on this site, grouped by what you came for — start here, dig into a specific workload, or read the engineering notes.
// start here
Features
→Lossless roundtrip, the Solana dictionary, six presets, feature-gated backends.
Presets
→All six presets side by side — level, ratio range, and when to reach for each.
Quickstart
→Cargo add, pick a preset, compress, decompress, assert the round-trip.
How it works
→The trait layer, the algorithms, and the dictionary that does the work.
FAQ
→Is it really 60:1? Is it lossless? Does it need a validator? Answered.
About
→Design notes — why Zstandard, why a hardcoded dictionary, why a trait layer.
// use cases by workload
Archive nodes
→Full history at up to 60:1 with MaxCompression — CPU you pay once at write time.
Account snapshots
→Dense repeated owner and program fields — the Accounts preset at 15–40:1.
Block explorers
→Keep more of history on warm disk with Mixed at an illustrative 12–35:1.
Indexer & DePIN
→Warehouse and replicate the firehose with Transactions at 10–30:1.
// from the blog
Reading the algorithm
→A source-level tour of the Solana preset — the dictionary, the Zstd construction, the level mapping.
Measuring ratios that matter
→How to benchmark compression honestly, and why the lower bound matters more than the headline.
Generic compressors & chain data
→What vanilla zstd actually does on Solana bytes, and what a hardcoded dictionary buys you.
// faq
Questions teams ask before they adopt.
+ Where does the 60:1 number come from?
The one measured figure we cite is "up to 60:1 on archival workloads with the MaxCompression preset" (Zstd level 19). It is the top of the range, not the median, and it assumes the input has the repetitive byte structure typical of account snapshots or batched archival data. Every other ratio here is an illustrative preset range, not a guaranteed result on your data.
+ How does this help with blockchain state bloat?
State bloat is a storage-and-bandwidth problem: high-throughput chains add state every slot and never reclaim it, so full history grows without bound. Compression does not shrink live consensus state, but it shrinks the archival and historical copies that archive nodes, indexers, and DePIN storage networks are paid to keep. On repetitive archival Solana data the MaxCompression preset reaches up to 60:1, flattening a storage curve that otherwise tracks block height.
+ Which chains are supported today?
The shipping preset is SolanaCompressor, which builds its dictionary from common Solana program IDs, instruction discriminators, and lamport amounts. The core trait layer is chain-agnostic — you can build your own dictionary for other chains — but the README only ships Solana presets.
+ Is the output guaranteed lossless?
Yes. Round-trip equality is enforced by tests on every preset. The README states "100% lossless — rigorous roundtrip validation on every operation." If decompress(compress(data)) != data, the library is failing a unit test.
+ Which backend should I enable?
Use the zstd feature for the Solana presets. DEFLATE and LZ4 are available as alternative backends behind their own feature flags, but the dictionary-driven presets that achieve the headline ratios require zstd.
+ Is the dictionary trained on my data?
No. The dictionary is hardcoded from a fixed list of common Solana byte patterns and built at compressor construction. Decompression is deterministic and the binary is self-contained — there is no online training loop and no per-customer state.
+ Does it depend on a running Solana node?
No. It is a pure compression library. You hand it bytes, it returns smaller bytes. No RPC, no validator, no network dependency.
// ship it
Turn state bloat into a smaller bill.
Add the crate, pick a preset that matches your workload, and let the dictionary collapse the repetitive structure your chain generates every slot. Round-trip the output. Confirm the ratio on your own data before you plan capacity around it.