# klauspost/compress **Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/klauspost-compress).** 5,525 stars · 375 forks · Go · NOASSERTION ## Links - GitHub: https://github.com/klauspost/compress - awesome-repositories: https://awesome-repositories.com/repository/klauspost-compress.md ## Topics `compression` `decompression` `deflate` `go` `golang` `gzip` `snappy` `zip` `zstandard` `zstd` ## Description This is a high-performance Go compression library providing implementations of Zstandard, Snappy, and Huffman coding. It includes a parallel compression framework for distributing gzip and stream workloads across multiple CPU cores and a specialized Huffman codec optimized for modern CPU architectures. The library features a Zstandard implementation that supports custom dictionaries and allocation-free decoding, alongside a Snappy compatible encoder for high-throughput data processing. It provides specific tools for dictionary generation and optimization to improve compression ratios for small or similar data blocks. Additional capabilities cover HTTP compression middleware for handling gzip and zstd response bodies, as well as security mechanisms to mitigate side-channel attacks through random jitter and response padding. The project also supports compressed stream seeking, block concatenation, and the creation of self-extracting archives. ## Tags ### Data & Databases - [High-Performance Compression Libraries](https://awesome-repositories.com/f/data-databases/high-performance-compression-libraries.md) — Provides high-performance Go implementations of Zstandard, Snappy, and Huffman coding. - [Dictionary-Based Compression](https://awesome-repositories.com/f/data-databases/dictionary-based-compression.md) — Uses pre-trained data samples as lookup tables to improve compression ratios for small or similar data blocks. - [Compressed Index Seeking](https://awesome-repositories.com/f/data-databases/compressed-index-seeking.md) — Creates a coordinate map of compressed offsets to allow random access without full stream decompression. - [Parallel Implementations](https://awesome-repositories.com/f/data-databases/data-compression-algorithms/gzip-compression/parallel-implementations.md) — Provides a parallelized implementation of Gzip to distribute workloads across multiple CPU cores. ([source](https://github.com/klauspost/compress#readme)) - [Small Data Compression Optimization](https://awesome-repositories.com/f/data-databases/data-compression-algorithms/small-data-compression-optimization.md) — Encodes and decodes payloads under 100KB using a non-streaming interface to avoid stream overhead. ([source](https://github.com/klauspost/compress/tree/master/s2)) ### Programming Languages & Runtimes - [Data Compression and Decompression](https://awesome-repositories.com/f/programming-languages-runtimes/data-compression-and-decompression.md) — Reduces data size for storage or transmission and restores it to its original form using optimized algorithms. ([source](https://github.com/klauspost/compress#readme)) - [Huffman Decoding](https://awesome-repositories.com/f/programming-languages-runtimes/data-compression-and-decompression/huffman-decoding.md) — The library decodes compressed blocks back into their original form using specialized decoding tables and stateless decoders. ([source](https://github.com/klauspost/compress/tree/master/huff0)) - [Snappy Compression](https://awesome-repositories.com/f/programming-languages-runtimes/data-compression-and-decompression/snappy-compression.md) — Implements S2 decompression that maintains compatibility with standard Snappy-encoded content. ([source](https://github.com/klauspost/compress/tree/master/s2)) - [S2 Extensions](https://awesome-repositories.com/f/programming-languages-runtimes/data-compression-and-decompression/snappy-compression/s2-extensions.md) — Provides S2 compression, a high-throughput extension of Snappy with concurrent processing support. ([source](https://github.com/klauspost/compress/tree/master/s2)) - [Zstandard Compression](https://awesome-repositories.com/f/programming-languages-runtimes/data-compression-and-decompression/zstandard-compression.md) — Implements the Zstandard algorithm with configurable settings to balance speed and compression ratios. ([source](https://github.com/klauspost/compress/tree/master/zstd)) - [Allocation-Free Decoders](https://awesome-repositories.com/f/programming-languages-runtimes/data-compression-and-decompression/zstandard-compression/allocation-free-decoders.md) — Offers an optimized Zstandard decoder that eliminates memory allocations during data restoration. ([source](https://github.com/klauspost/compress/tree/master/zstd)) ### Part of an Awesome List - [Compression Libraries](https://awesome-repositories.com/f/awesome-lists/data/compression-libraries.md) — A comprehensive collection of high-performance Go packages for data compression and decompression. - [Parallel Compression Tools](https://awesome-repositories.com/f/awesome-lists/devtools/file-compression/parallel-compression-tools.md) — Implements a parallel compression framework that utilizes multiple threads to increase stream throughput. ([source](https://github.com/klauspost/compress/tree/master/zstd)) - [Compression Utilities](https://awesome-repositories.com/f/awesome-lists/devtools/compression-utilities.md) — Optimized and high-performance compression packages. ### Operating Systems & Systems Programming - [Compression Dictionary Generation](https://awesome-repositories.com/f/operating-systems-systems-programming/compression-dictionary-generation.md) — Analyzes sample data sets to generate optimized dictionaries that improve compression ratios for common data formats. ([source](https://github.com/klauspost/compress/tree/master/dict)) - [Allocation-Free Decoding](https://awesome-repositories.com/f/operating-systems-systems-programming/allocation-free-decoding.md) — Implements decompression using pre-allocated buffers to eliminate garbage collection overhead during data restoration. ### Scientific & Mathematical Computing - [Huffman Coding Implementations](https://awesome-repositories.com/f/scientific-mathematical-computing/huffman-coding-implementations.md) — Implements a specialized Huffman codec optimized for modern CPU architectures to minimize byte counts. - [Huffman Codecs](https://awesome-repositories.com/f/scientific-mathematical-computing/huffman-coding-implementations/huffman-codecs.md) — Encodes independent data blocks using a Huffman codec optimized for modern CPUs to minimize byte counts for similar values. ([source](https://github.com/klauspost/compress/tree/master/huff0)) - [Finite State Entropy Coding](https://awesome-repositories.com/f/scientific-mathematical-computing/entropy-coding-algorithms/finite-state-entropy-coding.md) — Uses near-optimal symbol encoding to compress byte blocks based on the probability of value occurrences. ### Graphics & Multimedia - [Finite State Entropy Decompression](https://awesome-repositories.com/f/graphics-multimedia/image-decoding/entropy-decoders/finite-state-entropy-decompression.md) — Restores compressed byte blocks to their original form using the inverse of the entropy encoding process. ([source](https://github.com/klauspost/compress/tree/master/fse)) - [Finite State Entropy Compression](https://awesome-repositories.com/f/graphics-multimedia/image-encoding-libraries/arithmetic-entropy-encoding/finite-state-entropy-compression.md) — Compresses byte blocks using near-optimal symbol encoding to reduce size for inputs with many similar values. ([source](https://github.com/klauspost/compress/tree/master/fse)) ### Networking & Communication - [Payload Optimization](https://awesome-repositories.com/f/networking-communication/network-optimization/payload-optimization.md) — Improves bandwidth efficiency by optimizing the size of compressed request and response payloads. ### Security & Cryptography - [Side-Channel Attack Mitigations](https://awesome-repositories.com/f/security-cryptography/side-channel-attack-mitigations.md) — Prevents BREACH side-channel attacks by injecting random jitter and padding into compressed server responses. ([source](https://github.com/klauspost/compress/tree/master/gzhttp)) ### Software Engineering & Architecture - [Stateless Buffer Management](https://awesome-repositories.com/f/software-engineering-architecture/stateless-buffer-management.md) — Executes compression without maintaining internal state between write calls to reduce memory overhead for concurrent tasks. ### Web Development - [Compression Middleware](https://awesome-repositories.com/f/web-development/compression-middleware.md) — Provides middleware for automatically compressing and decompressing HTTP response bodies.