facebook/rocksdb
Rocksdb
RocksDB is a high-performance, embeddable persistent key-value library and storage engine based on Log-Structured Merge-trees. It is designed to provide durable storage for large-scale datasets, integrating directly into applications to manage data on flash and RAM-based hardware.
The engine is distinguished by its focus on minimizing read and write amplification through multi-threaded compaction and custom memory allocators. It features specialized optimizations for flash storage, including support for zoned block devices, and provides the ability to extend store behavior via external plugins.
Its broad capability surface includes atomic transactions, column family partitioning for logical keyspace division, and data-at-rest encryption. The system also supports secondary indexing, time-to-live data expiration, and integration with distributed filesystems.
Observability is provided through internal statistics tracking, component performance benchmarking, and crash recovery simulation.
Features
- Key-Value - Implements a high-performance storage engine that persists data as key-value pairs optimized for flash and RAM.
- Log-Structured Merge-Trees - Uses a Log-Structured Merge-Tree architecture to buffer writes in memory and flush them to sorted immutable files.
- Column Family Management - Organizes data into multiple logical partitions with independent settings and compaction strategies.
- Compaction Strategies - Provides configurable compaction strategies to balance write amplification and read efficiency in the LSM-tree.
- Data Partitioning - Divides the keyspace into independent namespaces with separate memtables and compaction settings for logical data organization.
- Atomic Transactions - Ensures consistency across multiple operations using optimistic and pessimistic atomic transaction patterns.
- Sorted String Tables - Persists data in immutable sorted files containing indices and bloom filters for fast point lookups and range scans.
- High-Performance Databases - Engineered for extreme write throughput and low-latency retrieval using multi-threaded compaction.
- Key-Value Persistence Stores - Provides a durable data store for large-scale datasets with atomic transactions and column families.
- Key-Value Stores - Serves as a persistent, embeddable key-value store integrated directly into applications.
- Large-Scale Dataset Management - Manages multi-terabyte datasets efficiently using multi-threaded compactions.
- Tiered Compaction Strategies - Merges smaller sorted files into larger ones across multiple levels to reclaim space and minimize write amplification.
- Write-Ahead Logging - Records every mutation to a sequential on-disk log to ensure data durability and recovery after crashes.
- Sorted Memory Buffers - Buffers incoming writes in a sorted memory structure (MemTable) before flushing them as immutable sorted string tables.
- Point-In-Time Snapshots - Generates consistent point-in-time snapshots of the database for backup or analysis purposes.
- Custom Grouped Operations - Defines custom merge operators to aggregate multiple values for a single key during reads or compaction.
- Flash Storage Managers - Manages data placement and write patterns to maximize SSD and zoned storage efficiency.
- Data Caching - Improves read throughput by storing frequently accessed data on local disks or SSDs.
- Data Compression Algorithms - Reduces the storage footprint of keys and values using compression algorithms to balance CPU and disk usage.
- Caching and Performance - Reduces latency and improves read throughput by caching frequently accessed data on local hardware.
- Compaction Filters - Applies custom logic to drop or transform keys and values while the database reorganizes internal files.
- Expiration Policies - Automatically removes old data based on a configurable time-to-live (TTL) threshold.
- Distributed System Metadata - Provides a durable and efficient storage layer for tracking metadata in large-scale distributed systems.
- In-Memory Caches - Implements a memory cache to store frequently accessed data and index blocks, reducing disk I/O latency.
- Key-Value Store Iterators - Provides forward and backward iterators for traversing sorted key-value pairs with snapshot isolation.
- Range Deletions - Removes all keys within a specified range using tombstone markers that are physically purged during compaction.
- Local State Stores - Enables fast access for stream processing by storing application state on local SSDs.
- Secondary Indexes - Provides secondary indexing capabilities to enable efficient lookups on columns other than the primary key.
- Storage Amplification Tuning - Allows tuning of write, read, and space amplification to optimize performance for specific hardware and workloads.
- Storage Integrity Validation - Verifies checksums for files and memtables to detect and report storage corruption.
- System Metadata Tracking - Tracks filesystem and object metadata for large-scale distributed systems across billions of entries.
- Plugin-Extensible Runtimes - Allows injecting custom logic via external plugins to modify internal storage engine operations.
- Custom Memory Allocators - Utilizes custom memory allocators to optimize heap usage and minimize resource contention during high-throughput operations.
- Zoned Block Device Support - Optimizes data placement for zoned storage devices to improve write efficiency and device lifespan.
- Language Bindings - Provides third-party bindings to connect the storage engine to various programming languages across different application stacks.
- Internal Telemetry Monitors - Monitors internal operational state including read/write latency, cache hit rates, and compaction times.
- Database Engines - Embeddable persistent key-value store for fast storage.
- Database Systems - Embedded key-value store optimized for fast storage.
- Datenbanken - High-performance embedded key-value store.
- Databases and Data - High-performance embedded key-value store.
