skyzh/mini-lsm
Mini Lsm
Mini-LSM is an educational storage engine and key-value database library designed to demonstrate the implementation of log-structured merge-tree architecture. It serves as a pedagogical resource for understanding how to build high-performance storage systems from the ground up, focusing on the mechanics of persistent data structures and disk-based storage.
The project provides a functional framework for managing data through memory-to-disk flushing and multi-version concurrency control. It distinguishes itself by implementing snapshot-based isolation, which allows for consistent views of the database state during concurrent operations, and utilizes leveled compaction strategies to organize data files and optimize read performance.
The engine covers a broad range of storage management capabilities, including write-ahead logging for crash recovery and probabilistic indexing to minimize disk input and output. It also includes a suite of verification utilities and testing tools intended to validate the integrity of storage operations and assist in diagnosing concurrency-related issues.
Features
- LSM-Tree Storage Engines - Implements a log-structured merge-tree database engine designed for learning key-value storage architecture.
- Log-Structured Merge-Trees - Implements a log-structured merge-tree architecture to manage data persistence through memory-to-disk flushing.
- Embedded Key-Value Stores - Provides an embedded storage framework that organizes data into sorted immutable files for efficient operations.
- Snapshot Isolation Techniques - Maintains data consistency during concurrent operations using snapshot isolation techniques.
- Compaction Strategies - Reorganizes and merges sorted data files to reclaim disk space and optimize read performance.
- Database Systems - Provides a framework for implementing core database components like logging, memory tables, and disk storage.
- Level-Based Compaction Strategies - Organizes data files into distinct levels and merges them to reclaim space and maintain read performance bounds.
- Multi-Version Concurrency Control - Maintains multiple versions of data records to enable non-blocking read and write operations.
- Bloom Filters - Uses probabilistic membership testing to quickly determine if a key exists in a file before performing disk I/O.
- Educational Database Systems - Serves as a pedagogical resource for learning the internals of storage engine implementation and database systems.
- Storage Optimizations - Applies compaction and merging strategies to improve disk space efficiency and read performance.
- Serializable Snapshot Isolation - Provides consistent views of the database state for long-running transactions using timestamp-based snapshot isolation.
- Write-Ahead Logging - Records incoming write operations to a persistent log to ensure durability and crash recovery.
- LSM Memtables - Buffers writes in sorted in-memory structures before flushing them as immutable files to persistent storage.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
