dgraph-io/badger
Badger
Badger is an embeddable key-value store written in Go that provides persistent data storage for byte keys and values. It is a persistent database that utilizes a tiered LSM tree storage model to optimize disk storage and retrieval efficiency.
The system features an ACID transaction engine that ensures data integrity through serializable snapshot isolation and multi-version concurrency control. It also provides an encrypted key-value store with data-at-rest encryption and a managed encrypted key registry to secure stored information.
The engine covers a broad set of capabilities including high-throughput data ingestion, persistent data indexing, and crash recovery via write-ahead logging. It supports advanced data management such as asynchronous value merging, background compaction, and prefix-based data streaming. Monitoring and observability tools are included for data integrity validation and performance metrics.
The project includes command-line utilities for performing offline database backup and restoration of the database state.
Features
- Tree-Based Storage - Utilizes a tiered Log-Structured Merge-Tree (LSM) architecture to optimize write throughput and read efficiency.
- Key-Value - Implements a high-performance embeddable database for persisting byte keys and values.
- ACID Transactional Cores - Provides a transactional core with serializable snapshot isolation to guarantee data integrity.
- Log-Structured Merge-Trees - Uses a tiered LSM tree storage model to optimize disk storage and retrieval efficiency.
- Compaction Strategies - Implements background compaction to merge sorted data levels and reclaim disk space.
- Storage Compaction Utilities - Reorganizes and merges key-value ranges to optimize storage efficiency and reduce disk fragmentation.
- Manifest-Based State Reconciliation - Reconciles on-disk table files with a manifest record to ensure data consistency and restore state.
- Embeddable Databases - Provides a persistent embeddable database library with built-in crash recovery and write-ahead logging.
- Sorted String Tables - Persists data using Sorted String Tables (SSTables) for efficient retrieval from immutable disk files.
- Sorted Indexing - Organizes data into sorted ranges to enable fast point lookups and efficient prefix scanning.
- Key-Value Persistence Stores - Provides a high-performance, embeddable Go library for persisting byte keys and values.
- Value-Log Separation - Reduces write amplification by storing keys in sorted tables and larger values in separate append-only logs.
- Multi-Version Concurrency Control - Implements MVCC to provide serializable snapshot isolation for concurrent read and write transactions.
- Persistent Application State - Ensures application state is persisted to disk and remains consistent across system restarts.
- Tiered Storage Strategies - Organizes data into multiple sorted levels to optimize storage and retrieval performance.
- Transaction Management - Manages transaction lifecycles and isolation levels to ensure concurrent read/write consistency.
- Write-Ahead Logging - Uses a write-ahead log to record changes for durability and crash recovery.
- Encrypted Persistence - Secures stored information on disk using provided encryption keys with support for key rotation.
- Encrypted Key Registries - Implements a persistent encrypted file for managing data keys with in-memory and read-only modes.
- Parallel Snapshot Iteration - Enables concurrent iteration over database snapshots using multiple workers to batch filtered key-value pairs.
- Point-In-Time Snapshots - Allows retrieving values using read-only transaction views to see data as it existed at a specific point in time.
- Data Integrity and Validation - Implements internal checks to ensure key ordering and index consistency across storage levels.
- Database Backups - Provides command-line utilities for performing offline database backups and restoration of state.
- Atomic Key-Value Operations - Enables atomic updates by combining multiple values via user-defined functions, avoiding read-modify-write cycles.
- High-Throughput Ingestion Pipelines - Optimizes high-throughput data ingestion using write-ahead logging and tiered LSM storage.
- Asynchronous Value Merging - Combines multiple values for a single key via a custom function and collapses them in the background.
- Prefix-Based Scanning - Supports reading key-value pairs filtered by specific key prefixes or selection functions.
- Key-Value Store Iterators - Provides utilities for traversing sorted key-value pairs sequentially for efficient range scans.
- Storage Engine Tuning - Provides settings to adjust memory table sizes and cache limits for optimized resource consumption.
- Custom Value Merging - Provides custom-function value merging to efficiently combine multiple values for a single key.
- Database Bootstrapping Streams - Accelerates restoration and replication by writing pre-sorted data streams directly to storage levels.
- Value-Log Reclamation - Reclaims disk space by removing expired or obsolete data from value log files through a rewrite process.
- Storage Integrity Verifiers - Validates checksums for entries in the value log and table blocks to detect data corruption.
- Encrypted Key-Value Stores - Offers data-at-rest encryption and a managed encrypted key registry to secure stored information.
- Database Performance Metrics - Exposes internal counters for read/write bytes and operation counts to monitor database health.
- Database Engines - Fast key-value store in Go.
- Database Systems - Listed in the “Database Systems” section of the Awesome Go awesome list.
- Database Tools - KV database with ACID transactions.
- Datenbanken - Fast, embedded key-value store.
- Databases and Storage - Fast key-value database.
- Databases & Data - High-performance embedded key/value store optimized for SSDs.
- Key-Value Stores - Embeddable, persistent, and fast key-value database.
