cockroachdb/pebble
Pebble
Pebble is an embedded key-value storage engine written in Go, designed as a library that provides durable, write-optimized data persistence directly within applications. It organizes data using a log-structured merge-tree (LSM-tree) structure, where writes are first buffered in an in-memory skiplist memtable and persisted to a write-ahead log before being flushed to block-based SSTable files on disk. The engine supports atomic batch commits, configurable write synchronization, and automatic background compaction that merges and rewrites sorted runs to reclaim space and maintain read performance.
The storage engine distinguishes itself through support for range keys, which allow key-value pairs to apply to contiguous ranges of keyspace and be interleaved with point keys during iteration. It also includes property-based filtering, where user-defined key-value properties attached to SSTable blocks and files enable iteration to skip irrelevant data blocks and entire files during scans. Pebble provides a storage format migrator that can upgrade on-disk database files to newer physical formats at runtime, using either background or blocking compaction to apply the change.
The engine offers comprehensive key-value operations including point lookups, range scans in both forward and reverse directions, and batch write operations that group multiple mutations into a single atomic commit. It includes built-in benchmarking tools for measuring read, write, and mixed workload throughput and latency under realistic access patterns, using configurable key distributions and value sizes.
Features
- Embedded Database Engines - Provides an embedded key-value storage engine that runs directly within applications without external server processes.
- Log-Structured Merge-Trees - Organizes data using a log-structured merge-tree with writes buffered in memory and flushed to sorted immutable files.
- Key-Value Stores - Reads stored values by key, returning the associated data and a handle for resource cleanup.
- LSM-Tree Batch Commits - Provides atomic batch commits that group mutations into a single unit applied to the memtable and WAL.
- LSM-Tree Batch Writes - Atomically groups multiple mutations into a single commit to reduce write amplification in the LSM-tree.
- Write-Optimized Storage Engines - Designed for high write throughput using batched atomic commits and background compaction to reduce write amplification.
- Automatic Background Compactions - Automatically merges and rewrites sorted runs in the background to reclaim space and maintain read performance.
- Block-Based SSTable Formats - Implements a block-based SSTable format with index and bloom filter blocks for efficient point lookups and range scans.
- Level-Based Compaction Strategies - Automatically merges and rewrites sorted runs across multiple levels in the background to reclaim space.
- Skiplist Memtables - Uses an in-memory concurrent skiplist memtable to buffer recent writes before flushing to disk.
- Compaction Management - Automatically merges and rewrites sorted runs in the background to reclaim space and maintain read performance.
- Key-Value - Persists key-value data to disk using an LSM-tree structure with atomic writes and configurable sync behavior.
- Embedded Key-Value Stores - Provides an embedded key-value storage engine with LSM-tree organization, atomic batch writes, and configurable durability.
- LSM-Tree Key-Value Stores - Implements a full LSM-tree key-value store with atomic batch commits, write-ahead logging, and background compaction.
- Write-Ahead Logging - Persists every write to an append-only log before applying it to the memtable for durability and crash recovery.
- Indexed Batch Writes - Combines multiple mutations into a single indexed batch for efficient, concurrent commit processing.
- Storage Engine Benchmarks - Ships built-in benchmarking tools for measuring read, write, and mixed workload performance under configurable key distributions.
- Physical Format Migrations - Upgrades a database to newer physical file formats at runtime using background or blocking compaction.
- Range Key Managers - Defines key-value pairs over a range of keyspace with custom semantics, interleaving them during iteration.
- Range Key Iteration - Defines and iterates over key-value pairs that apply to a range of keyspace with custom semantics and interleaved iteration.
- Range Key Iterators - Supports range keys that apply to contiguous key ranges and interleaves them with point keys during iteration.
- Reverse Key Iterators - Traverses key-value pairs in descending order using backward links in the memtable's skiplist.
- Property-Based Filtering - Attaches user-defined key-value properties to SSTable blocks and files to skip irrelevant data during iteration.
- SSTable Block Filters - Implements property-based filtering that skips irrelevant SSTable blocks and files during iteration to accelerate queries.
- Runtime Storage Format Upgrades - Migrates a database to newer physical file formats at runtime using background or blocking compaction.
- Caching Libraries - KV storage engine.
- Database Engines - RocksDB/LevelDB-inspired key-value database.
- Database Systems - Listed in the “Database Systems” section of the Awesome Go awesome list.
- Storage Systems - Key-value store optimized for high-performance database engines.
