Explore open-source projects that implement custom key-value storage engines inspired by the Redis data structure architecture.
KeyDB is a multithreaded in-memory key-value store and distributed cache. It functions as a NoSQL database utilizing multi-version concurrency control to execute non-blocking queries and scans. The project is a multithreaded fork of Redis that maintains protocol compatibility while utilizing a multithreaded architecture to scale across multi-core hardware. It distinguishes itself with flash-tiered storage, allowing the system to offload data from primary RAM to SSD or flash storage to increase total capacity. The system supports high availability through active-active mesh replication and multi-master synchronization. Additional capabilities include cloud-native snapshotting for backups, Transport Layer Security for encrypted communication, and subkey-level expiration for nested data structures. The server can be operated as a background daemon with configuration managed via startup arguments or files, and it includes a dedicated command-line interface for data management.
KeyDB is a high-performance, multithreaded in-memory key-value store that maintains full RESP protocol compatibility while offering advanced features like active-active replication and tiered flash storage.
Valkey is an in-memory, NoSQL database server designed for high-performance data storage and real-time state management. It operates as a distributed key-value store, maintaining datasets entirely within system memory to facilitate sub-millisecond response times for read and write operations. The system distinguishes itself through a single-threaded event loop that utilizes asynchronous I/O multiplexing to ensure high throughput. It supports high availability via master-replica replication and provides a decoupled communication model through a built-in publish-subscribe messaging pattern. To ensure data durability, the engine employs a copy-on-write mechanism to generate point-in-time snapshots of the dataset on disk. The platform offers extensive infrastructure customization, allowing users to compile binaries from source with specialized memory allocators and hardware-level configurations. These capabilities enable the deployment of scalable, distributed storage clusters tailored to specific performance and hardware requirements.
Valkey is a high-performance, in-memory key-value store that provides the exact architecture and feature set requested, including persistence, replication, and support for the RESP protocol.
Dragonfly is a high-performance, multi-model in-memory data store designed to serve as a drop-in replacement for existing database infrastructures. By utilizing a multi-threaded, shared-nothing architecture and a fiber-based concurrency model, it maximizes CPU utilization and minimizes latency for read and write operations. The system supports a wide range of data structures, including strings, hashes, lists, sets, sorted sets, and JSON documents, while maintaining full compatibility with standard industry wire protocols and client libraries. What distinguishes Dragonfly is its focus on efficiency and scalability through advanced memory management and request processing. It employs a lock-free, cache-friendly hash table structure and zero-copy serialization to reduce overhead during high-throughput operations. For durability, the system utilizes asynchronous, snapshot-based persistence that captures the state of the dataset without blocking active requests. Furthermore, it provides built-in support for horizontal scaling and cluster management, allowing for the distribution of large datasets across multiple nodes to ensure high availability. Beyond core storage, the platform includes a comprehensive suite of operational and analytical capabilities. It features integrated support for geospatial data management, real-time message brokering via publish-subscribe patterns, and full-text search. To handle massive datasets efficiently, the engine incorporates probabilistic data structures for cardinality estimation, frequency tracking, and membership testing. These features are complemented by robust administrative tools, including access control, request rate limiting, and detailed server monitoring.
Dragonfly is a high-performance, multi-threaded in-memory data store that implements the RESP protocol and provides the persistence, distributed capabilities, and advanced data structures required for a modern key-value store.
Redis is an in-memory, key-value database designed to provide sub-millisecond latency for read and write operations. It functions as a versatile data platform, serving as a distributed cache, a message broker, a NoSQL document store, and a vector database. The system utilizes an event-driven, single-threaded loop to process requests efficiently, while maintaining data durability through append-only persistence logs and asynchronous snapshotting mechanisms. What distinguishes Redis is its ability to handle complex data structures—including strings, hashes, lists, sets, and sorted sets—alongside hierarchical JSON documents and high-dimensional vector embeddings. It supports advanced operational patterns such as active-active database deployment for global distribution, real-time data streaming, and probabilistic statistics for large-scale data analysis. These capabilities are complemented by a pluggable indexing engine that enables semantic similarity matching and full-text retrieval. The platform offers a comprehensive ecosystem for managing distributed state, including master-replica replication, automated cluster management, and granular security controls like access control lists and TLS encryption. Developers can interact with the database through language-specific client libraries that support connection multiplexing and object mapping, or via a command-line interface for direct administrative tasks and scripting. Redis is deployed through standard package managers and supports both self-managed clusters and managed cloud instances. Observability is provided through integrated tools for performance analysis, slow log monitoring, and bulk data management.
This is the industry-standard, high-performance in-memory key-value store that serves as the primary reference implementation for the RESP protocol, multi-threaded-adjacent event loops, and robust persistence mechanisms.
Dice is a low-latency key-value store based on Valkey that functions as a hierarchical storage engine. It combines a high-speed memory store with a tiered cache system to manage data across multiple storage levels. The system extends memory capacity by implementing a disk-backed cache that automatically persists evicted items to disk and restores them during cache misses. This storage tiering balances retrieval speed and operational cost by distributing data between fast memory and slower disk storage. The engine includes a query subscription service that monitors specific data patterns and provides real-time notifications whenever matching information changes.
Dice is a high-performance, in-memory key-value store that provides Redis-compatible functionality while implementing advanced tiered storage and real-time data subscription features.
Redis is a high-performance in-memory key-value store that functions as a distributed cache, message broker, and NoSQL database. It provides sub-millisecond read and write access to data stored in RAM and can operate as a vector database for indexing high-dimensional embeddings. The system supports a wide range of data storage and synchronization primitives, including the management of strings, hashes, lists, sets, and JSON documents. It enables real-time data operations through atomic transactions, hybrid persistence using snapshots and append-only logs, and high-availability configurations such as automated failover and geographic data distribution. Capabilities extend to asynchronous messaging via publish-subscribe frameworks and event streams with consumer group coordination. The platform also includes advanced search and indexing for full-text, geospatial, and vector similarity queries, as well as tools for AI memory management and machine learning feature serving. The software can be deployed natively on Windows as a process or service, or within containerized environments like Kubernetes.
This is the official Redis repository, which serves as the industry-standard implementation of a high-performance, in-memory key-value store featuring the RESP protocol, multi-threaded architecture, and robust persistence mechanisms.
Tile38 is an in-memory geospatial database that uses the Redis protocol for communication and query execution. It serves as a distributed spatial store for points and polygons, featuring high availability through leader-follower replication and disk-based persistence. The system includes a real-time geofencing engine that monitors virtual geographic boundaries and triggers webhooks when objects enter or exit specified areas. It further distinguishes itself with an embedded Lua scripting engine for server-side data transformations and a built-in vector tile server for efficient map visualization. The platform supports diverse geospatial formats and provides spatial querying capabilities, including proximity searches, containment checks, and k-nearest neighbors. Data retrieval can be further refined using attribute and expression-based filtering, while object lifetimes are managed via time-to-live values. The server can be deployed using containerized images and supports secure client access through TLS encryption and role-based access control.
Tile38 is a high-performance, in-memory key-value store that implements the Redis protocol and includes persistence and replication, making it a relevant example for studying these architectural patterns despite its specialized focus on geospatial data.
DiceDB is a distributed cache manager and hierarchical storage engine based on a Valkey key-value store. It functions as a high-availability key-value cluster that balances performance and cost by distributing data across memory and disk tiers. The system supports working sets larger than available RAM by persisting evicted keys to disk and restoring them on demand. It also includes a real-time query subscription system that notifies clients when data matching specific criteria is updated. The project provides distributed cache coordination through master node health monitoring and quorum-based failover. Additional capabilities include gossip-based network discovery, a module extension system for loading shared objects, and tools for managing process logging and network interface binding.
DiceDB is a high-performance, in-memory key-value store that implements the RESP protocol and features multi-threaded architecture, disk-backed persistence, and distributed coordination, making it a comprehensive example of the requested technology.
This project is a native Windows port of the Redis in-memory key-value store. It provides a high-speed database that maintains datasets in RAM while using periodic snapshots to disk for data persistence. The implementation includes a background system service wrapper, allowing the data store to be installed as a Windows service that starts automatically upon computer boot. It utilizes native Windows compilation to enable execution directly on the host operating system. The system covers in-memory data caching and local data persistence, ensuring that information is recovered after a process restart or system stop. It also handles operating system specific path resolution for storing data snapshots in valid local folders.
This repository is a platform-specific port of an existing database rather than an educational project or implementation guide for building a key-value store from scratch.
Bigcache is a thread-safe, in-memory key-value store for Go designed to minimize garbage collection pauses. It functions as a memory-optimized cache capable of storing gigabytes of data while maintaining low latency during high-volume operations. The system avoids garbage collection overhead by managing memory through large byte arrays and manual allocation. It utilizes a circular-buffer layout to reuse space and lock-striped sharding to reduce contention during concurrent read and write access. The project provides capabilities for high-volume in-memory caching and low-latency data retrieval, specifically targeting the management of massive datasets within distributed system environments.
Bigcache is a high-performance, in-memory key-value store designed for low-latency caching in Go, though it functions as a specialized library for memory management rather than a full-featured, protocol-compatible database server.
Ristretto is a high-performance in-memory cache and concurrent key-value store for Go applications. It provides a thread-safe memory store that manages strict memory bounds and employs probabilistic set filters to reduce lookup overhead. The system is distinguished by an admission-policy cache that utilizes frequency sketches and cost-based eviction to maximize hit ratios. It minimizes contention and improves throughput through the use of striped ring buffers and concurrent map sharding. The project covers a broad range of data management capabilities, including time-based expiration, item frequency estimation, and batch cache updates. It also includes operational telemetry for tracking cache performance metrics and hit ratios, along with the ability to serialize filter states.
This is a high-performance, concurrent in-memory key-value store designed as a library for Go applications, providing a practical look at advanced cache eviction and memory management techniques.