Open-source alternatives to Mini Lsm

Pebble is an embedded key-value storage engine written in Go, designed as a library that provides durable, write-optimized data persistence directly within applications. It organizes data using a log-structured merge-tree (LSM-tree) structure, where writes are first buffered in an in-memory skiplist memtable and persisted to a write-ahead log before being flushed to block-based SSTable files on disk. The engine supports atomic batch commits, configurable write synchronization, and automatic background compaction that merges and rewrites sorted runs to reclaim space and maintain read performanc

LevelDB is an embedded database library and persistent storage engine that provides a sorted key-value store. It uses a log-structured merge-tree architecture to map byte arrays to values, running directly within a process to provide storage without the need for a separate server process. The system is distinguished by its use of custom comparison functions to define key ordering, enabling efficient range scans and sequenced lookups. It ensures data reliability through atomic batch execution, consistent snapshot generation, and log-based recovery after failures. The engine covers broad capab

Sled is an embedded key-value store and ACID-compliant database designed for high-performance data persistence. It functions as a log-structured storage engine that organizes data using B+ trees to support efficient range queries and prefix scans. The engine implements a zero-copy data store model, utilizing epoch-based reclamation to provide direct references to cached values without memory allocations. It distinguishes itself through a combination of write-ahead logging, page cache optimizations to reduce write amplification on flash storage, and serializable transactions for atomic multi-k

Kvrocks is a distributed key-value store and Redis-compatible NoSQL database. It utilizes a RocksDB storage engine to provide disk-based persistence, allowing for high-capacity data storage with reduced memory costs compared to in-memory systems. The system functions as a vector database and full-text search engine, supporting nearest-neighbor searches on vector embeddings and complex document queries via text matching. It employs a proxyless cluster architecture with slot-based routing to distribute data and scale capacity across multiple nodes. The platform covers a wide range of data mana

Kvrocks is a disk-based NoSQL database and distributed key-value store that leverages the RocksDB storage engine to persist large datasets to physical disk. It is designed to be a Redis-compatible database, utilizing the standard Redis communication protocol to ensure interoperability with existing client libraries and tools. The project distinguishes itself by combining a disk-persistent storage model with advanced retrieval capabilities, including vector search for k-nearest neighbor queries, full-text search indexing, and geospatial query execution. It supports distributed clustering with

Cassandra is a distributed NoSQL database and wide-column store designed for high availability and linear scalability. It functions as a fault-tolerant distributed system that utilizes an LSM-tree storage engine to optimize write throughput and manage massive datasets. The system is a CQL-compliant database, using a structured query language to manage and retrieve tabular data stored across multiple nodes. It organizes information into rows and columns based on a flexible schema and primary keys. The project provides capabilities for horizontal database scaling, distributed data partitioning

H2 is a JDBC-compliant relational database management system written in Java. It functions as an embeddable SQL database that can run directly within an application process to remove network latency, or as an in-memory database for high-performance volatile storage. It also includes a web-based console for executing SQL commands and administering schemas. The system is characterized by its flexible deployment modes, including a standalone server mode for remote TCP/IP access and a mixed mode for simultaneous local and remote connectivity. It features a dialect emulation layer and compatibilit

RocksDB is a high-performance, embeddable persistent key-value library and storage engine based on Log-Structured Merge-trees. It is designed to provide durable storage for large-scale datasets, integrating directly into applications to manage data on flash and RAM-based hardware. The engine is distinguished by its focus on minimizing read and write amplification through multi-threaded compaction and custom memory allocators. It features specialized optimizations for flash storage, including support for zoned block devices, and provides the ability to extend store behavior via external plugin

Badger is an embeddable key-value store written in Go that provides persistent data storage for byte keys and values. It is a persistent database that utilizes a tiered LSM tree storage model to optimize disk storage and retrieval efficiency. The system features an ACID transaction engine that ensures data integrity through serializable snapshot isolation and multi-version concurrency control. It also provides an encrypted key-value store with data-at-rest encryption and a managed encrypted key registry to secure stored information. The engine covers a broad set of capabilities including hig

This is an educational relational database engine used in Carnegie Mellon University's database systems course. Students learn internals by implementing core components of a working database, including storage, indexing, concurrency control, and crash recovery. The system covers key database architecture: a B+ tree index for fast key-based lookups and range scans, a disk-oriented buffer pool that caches pages from disk, an iterator-based query execution model that composes physical operators, page-based storage for records, two-phase locking for coordinating concurrent transactions, and write

This project is an open source relational database management system and SQL database designed for storing and managing structured data. It functions as a relational database for ensuring consistency and reliability, while also operating as a vector database for storing and querying high-dimensional vector embeddings. The system incorporates a columnar storage engine to optimize analytical query processing and large-scale data aggregation. It further enables vector similarity search, allowing users to find similar items by querying vector embeddings. The software covers a broad capability su

Sonic is a high-performance, lightweight search backend designed to provide real-time full-text search and autocomplete capabilities for applications. It functions as a persistent indexing server that maps text terms to object identifiers, allowing developers to integrate rapid search functionality without storing raw document content directly within the search engine. The system distinguishes itself through a specialized graph-based index that enables real-time word prediction and typo correction. Communication is handled via a custom, low-latency binary protocol over raw TCP sockets, which

MySQL Server is a relational database management system designed to organize and store structured information. It functions as a comprehensive SQL server platform that provides reliable transactional integrity and high-performance query execution for enterprise data management. The system distinguishes itself through a pluggable storage engine architecture that decouples logical query processing from physical data storage, allowing for specialized handling of diverse workloads. It maintains data consistency and high concurrency through multi-version concurrency control and write-ahead logging

etcd is a distributed key-value store and configuration store designed to maintain a consistent set of data across a cluster of nodes. It functions as a reliable registry for storing and synchronizing critical settings and metadata used by distributed applications. The system implements the Raft consensus algorithm to ensure data consistency and leader election across servers. To protect data transfers and verify node identities, it utilizes a network security layer based on mutual TLS and client certificates. Its capabilities cover distributed configuration management, cluster state synchro

PalDB is an embeddable key-value storage engine for Java designed for high-performance data retrieval. It functions as a library that persists data into a single, immutable binary file, providing a storage solution for write-once workloads that require minimal memory overhead compared to standard in-memory collections. The system distinguishes itself through a write-once data layout that eliminates the need for locking mechanisms during concurrent read operations. By utilizing memory-mapped file access and sorted key indexing, the engine enables rapid lookups directly within the process addre

HBase is a distributed, wide-column NoSQL store and big data storage engine designed for sparse datasets. It functions as a scalable columnar database built on top of the Hadoop Distributed File System to provide real-time read and write access to massive volumes of structured and unstructured data. The system acts as a cross-language database gateway, offering connectivity through native remote procedure calls, REST, and Thrift interfaces. It distinguishes itself through a master-worker coordination model that enables horizontal scaling and fault tolerance across a cluster. The project cove

NutsDB is an ACID-compliant, embedded transactional storage engine that functions as both a disk-backed key-value store and an in-memory data structure store. It provides atomic and serializable transactions with commit and rollback capabilities to ensure strict data consistency for applications requiring a lightweight persistence layer. The engine distinguishes itself by supporting a variety of complex data types, including lists, sets, and sorted sets, alongside standard byte-slice storage. It implements a transactional storage model featuring hot backups and a compaction algorithm to maint

BuntDB is an embedded key-value store for Go applications, providing in-memory storage with optional disk persistence. It structures data using a B-tree for ordered key-value access and an R-tree for spatial indexing, allowing both range scans and geometric intersection queries. Support for indexing on nested JSON document fields enables efficient lookups by values within JSON objects, and per-key time-to-live (TTL) expiration automatically removes stale entries. The store uses copy-on-write transaction isolation, ensuring each transaction sees a consistent snapshot and changes are applied at

Scylla is a distributed wide column NoSQL database designed as a high-performance data store. It functions as a Cassandra compatible database and a DynamoDB compatible store, implementing a shared-nothing architecture built on an asynchronous event-driven framework. The system emulates cloud-based APIs to support applications built for proprietary cloud protocols and implements the Cassandra Query Language for high-throughput workloads. This allows for the migration of cloud workloads to self-hosted environments while maintaining API compatibility. The project covers distributed data storage

RoseDB is a persistent key-value database and log-structured storage engine. It functions as a lightweight storage system that utilizes a log-structured hash table and a Bitcask engine implementation to provide fast data retrieval and disk-backed persistence. The system operates as an atomic transaction engine, grouping multiple read and write operations into single units to maintain data consistency. It handles data through a key-value model that supports individual insertions, lookups, and deletions. The database provides capabilities for batch data processing and atomic updates. Additiona

RoseDB is an embedded key-value database and log-structured storage engine. It functions as a library-based database that runs within an application process to provide local data persistence without the need for a separate database server. The project implements a Bitcask-style store, utilizing an append-only log format to maximize write throughput and minimize seek time. It ensures data consistency by grouping multiple read and write operations into single atomic units. The storage architecture relies on an in-memory key index for constant-time lookups and uses segment-based file management

pg_textsearch is a full-text search integration for PostgreSQL that provides large-scale text indexing and BM25 relevance ranking. It implements a scalable indexing architecture that uses a memtable system to spill data to disk segments, allowing for the processing of massive datasets. The project distinguishes itself through support for multilingual search via language-specific partial indexes and the ability to index complex expressions, such as JSONB fields or concatenated columns. It ensures high availability by utilizing PostgreSQL-native streaming replication and write-ahead logs to syn

Level is a database library that provides a unified interface for managing sorted key-value data. It functions as an abstraction layer that allows applications to store and retrieve binary information consistently across server-side environments and web browsers. The project utilizes a modular architecture that supports pluggable storage backends, enabling the system to adapt to different host environments while maintaining identical behavior. By organizing data in lexicographical order, it facilitates efficient range queries and ordered retrieval. The library handles large datasets through a

LMDB is an embedded key-value storage engine that provides ACID-compliant data persistence. It is a memory-mapped database that utilizes B+ trees to store key-value pairs, ensuring atomicity, consistency, isolation, and durability. The engine maps files directly into the virtual address space to minimize data copying and system calls. This approach enables high-performance local caching and low-latency data access, specifically optimizing for read-heavy database workflows. The system implements a transactional model with copy-on-write versioning and single-writer multi-reader locking. These

RisingWave is a cloud-native streaming database and real-time analytics engine that uses standard SQL to process continuous data streams. It functions as a streaming data lakehouse, combining the capabilities of a streaming SQL database with a platform that integrates streaming ingestion with open table formats. The system is distinguished by its use of the PostgreSQL wire protocol, allowing it to integrate with existing SQL tools and drivers. It employs a decoupled compute and storage architecture, persisting streaming state and materialized views in cloud object storage to enable independen

Redka is a SQL-backed key-value store that implements a Redis-compatible interface. It acts as a relational data bridge, translating non-relational protocol commands into relational queries to provide persistence and ACID transactions using a SQL database backend. The system allows for relational data introspection by exposing internal key-value data as SQL views, enabling direct querying and analysis via relational syntax. It can be deployed as a standalone network server or as an embedded library within a Go application. The project supports a variety of data structures, including strings,

Talent Plan provides guided training programs and curricula centered on distributed database design, systems programming, and open source contribution workflows. The project offers a distributed systems education program consisting of curated courses and labs focused on database internals. The curriculum emphasizes the use of the Rust language for building high-performance networked applications and implementing distributed algorithms. It integrates educational materials on version control, community governance, and the specific processes required to contribute to public software projects. T