facebook/zstd
Zstd
Zstandard is a lossless data compression library and archive format designed for high compression ratios and fast real-time processing. It functions as a real-time data compressor and multi-threaded compression engine capable of distributing workloads across multiple CPU cores to increase throughput.
The system features a dictionary-based compressor that trains on sample data to improve the compression ratio and speed of small files. It also provides long distance pattern matching to identify repeated sequences across large files.
The library covers a broad range of capabilities including stream and single file compression, high-throughput decompression, and the ability to perform pattern matching directly within compressed archives without full decompression. It includes tools for performance benchmarking and parameter tuning to balance processing speed and file size.
The project supports compilation into shared or static libraries and provides interfaces to replace standard compression libraries.
Features
- Data Compression Algorithms - Restores compressed data to its original form while supporting legacy formats from earlier versions.
- Real-Time - Reduces the size of data streams in real-time to balance high throughput and efficient compression ratios.
- Real-Time Compression - Offers real-time compression algorithms that balance immediate throughput with high compression ratios for data streams.
- Real-Time Decompression - Enables high-throughput real-time decompression to restore data quickly for immediate application use.
- Compression Libraries - Provides a comprehensive lossless data compression library with high compression ratios and fast processing speeds.
- Data Formats - Implements a stable and documented lossless archive format for storing compressed data.
- Dictionary-Based Compression - Features a dictionary-based compressor that trains on sample data to improve compression ratios for small files.
- Real-Time Data Compressors - Functions as a high-throughput real-time data compressor designed for performance-critical streaming applications.
- Compression Dictionary Training - Analyzes sample data to create dictionaries that improve compression ratios and speeds for small data sets.
- Sliding Window Matching - Identifies repeated data sequences across a sliding window of source bytes to replace duplicates.
- Compression Dictionary Generation - Creates dictionaries from sample data to improve the compression ratio for small or similar files.
- Compression Library Integrations - Provides interfaces to replace zlib compression, increasing data throughput and reducing file sizes.
- Dictionary-Based Compression - Improves ratios for small files by using a pre-trained table of common patterns.
- Dictionary-Based Decompression - Recovers data that was compressed using a specific dictionary to ensure accurate reconstruction.
- Multi-Threaded Compression Engines - Ships a multi-threaded compression engine capable of distributing workloads across multiple CPU cores.
- Parallel Block Processing - Distributes independent chunks of data to multiple CPU cores for simultaneous compression or decompression.
- Entropy Coding Algorithms - Implements finite state entropy to encode symbols with high throughput and minimal memory footprint.
- Sequence Matching Algorithms - Replaces recurring strings of bytes with references to previous occurrences within the data stream.
- File Compression - Provides utilities for compressing and reducing the size of individual files using simple or streaming modes.
- Small - Improves compression ratios for small files by training a dictionary on representative data samples.
- Compressed Data Searching - Performs pattern matching directly on compressed archives without requiring a full decompression step.
- Small Data Compression Optimization - Improves compression ratios for small files or records by training and applying dictionaries.
- High-Throughput Decompression - Provides a decompression engine optimized for high throughput to ensure high-speed data access.
- Long Distance Pattern Matching - Implements long distance pattern matching to identify repeated sequences across large files for increased compression ratios.
- Compression Frame Formats - Wraps compressed blocks in a structured format containing magic numbers and checksums to ensure data integrity.
- Compression Context Reuse - Minimizes allocation overhead by resetting and reusing existing buffers and state structures across multiple data streams.
- Context Reuse Strategies - Minimizes memory overhead by reusing existing compression contexts and resources to reduce frequent allocations.
- Data Stream Compression - Implements native data stream compression to optimize the size of continuous data flows.
- Throughput Optimizations - Increases speed and ratio by pledging the expected size of source data to the compressor for better tuning.
- Compression Parameter Tuning - Adjusts internal compression levels using predefined tables to balance speed and ratio.
- Performance Benchmarking - Includes tools to measure compression ratios and speeds using in-memory data for configuration evaluation.
- Compression Libraries - Fast real-time compression algorithm.
- Data Compression - Provides high-performance real-time data compression capabilities.
- Terminal Utilities - Fast real-time data compression algorithm.
