Embedded Key-Value Stores for Local Persistence

redb is an embedded key-value store and ACID-compliant storage engine. It functions as a persistent storage system for saving and retrieving data as key-value pairs within a tree structure. The engine is built as an MVCC transactional database, utilizing multi-version concurrency control to manage simultaneous reads and writes without blocking. It employs a single-writer multi-reader model to ensure data consistency while allowing multiple threads to access the store. The system provides persistent state management and atomic transaction management to prevent data corruption during crashes. It handles concurrent data access and ensures that groups of changes are applied as single units.

Sled is an embedded key-value store and ACID-compliant database designed for high-performance data persistence. It functions as a log-structured storage engine that organizes data using B+ trees to support efficient range queries and prefix scans. The engine implements a zero-copy data store model, utilizing epoch-based reclamation to provide direct references to cached values without memory allocations. It distinguishes itself through a combination of write-ahead logging, page cache optimizations to reduce write amplification on flash storage, and serializable transactions for atomic multi-key updates. The library covers a broad range of capabilities, including crash recovery through checkpointing, disk storage defragmentation, and binary stream replication. It also provides reactive data observation via key-prefix event subscriptions and supports custom merge logic for atomic value transformations.

Bolt is a single-file embedded key-value store for Go applications. It is an ACID transactional database that organizes data in B+trees on disk to provide efficient sorted key retrieval and range scans. The system uses a memory-mapped model to map the database file directly into the process address space for fast random-access reads. The project distinguishes itself through a multi-version concurrency control architecture that allows multiple simultaneous readers to access a consistent snapshot of data without blocking a writer. It employs a single-writer multi-reader locking model and uses advisory filesystem locking to prevent multiple processes from opening the same database file simultaneously. The database provides hierarchical data organization through nested buckets and supports unique identifier generation for records. It includes capabilities for hot backups, batch write updates to reduce disk commit overhead, and cursor-based key seeking and iteration. Operational telemetry is available via internal database performance metrics to track system activity.

Lowdb is a lightweight local data persistence library that manages application state as a JSON document. It functions as a database utility that stores information directly in local files or browser storage, removing the requirement for a dedicated database server or complex infrastructure. By treating stored data as standard objects and arrays, it provides a straightforward interface for managing application information. The library distinguishes itself through a modular adapter-based architecture that decouples data manipulation from the underlying storage medium. This design allows for the implementation of custom storage backends, including support for data transformations like encryption or compression. It also maintains a complete copy of the database in volatile memory, which enables high-speed operations and facilitates the use of in-memory storage modes to accelerate automated test suites. Beyond basic storage, the project supports data integrity through atomic write serialization, which ensures that the entire state is saved as a single operation to prevent corruption. It also provides mechanisms for schema validation, allowing developers to check database content against predefined structures to maintain consistent data formats. The library can be extended by integrating third-party utility libraries for complex queries or by linking database adapters to web servers to expose data through network endpoints.

LiteDB is a serverless, embedded NoSQL document database for .NET applications. It persists data into a single portable file, functioning as a BSON data store that resides within the application process rather than running as a separate server. The system is ACID compliant, utilizing write-ahead logging to ensure atomic, consistent, isolated, and durable transactions. It includes built-in encryption to provide secure local data storage and protect files on disk from unauthorized access. The project covers object-document mapping to convert classes into document formats, indexed search capabilities via B-tree indexing, and specialized streaming for large binary objects. It also provides a dedicated administrative studio for visual data administration and modification.

Epicenter is a local-first knowledge management system and data orchestrator designed to structure information generated by large language models into validated schemas. It functions as a storage architecture that persists application data in human-readable files and databases to ensure user ownership and portability. The system distinguishes itself by projecting language model outputs into structured, schema-validated tables and utilizing conflict-free replicated data types to synchronize application state across multiple devices without a central server. This allows for offline access and consistent state management while maintaining a decoupled content model where machine-generated outputs remain separate from user-curated folders. The platform covers a broad range of capabilities including API integration with local or remote text models, the definition of typed data schemas for consistency, and file-system-based persistence for long-term data portability.

SurrealDB is a multi-model database engine designed to store and query document, graph, relational, and vector data within a single ACID-compliant platform. It functions as an AI-native data store, integrating vector search, graph traversal, and machine learning model execution directly into its query layer. By providing a unified declarative query language, the platform eliminates the need for external middleware to synchronize data across different storage models. The platform distinguishes itself through its ability to manage agent memory and complex workflows natively. It allows developers to store agent memory, knowledge graphs, and structured data within a single transaction boundary, ensuring consistent state and permissions. Furthermore, the engine supports real-time reactive applications by pushing data updates directly to connected clients through live queries, removing the requirement for external message brokers or polling mechanisms. SurrealDB is built for versatility, operating as a portable database runtime that maintains a consistent interface across embedded, edge, and cloud environments. Its architecture includes a granular, record-level permission model that enforces security and multi-tenant isolation directly at the data layer. The system also features an isolated sandboxing environment for custom extensions, allowing for specialized data processing without compromising system stability or security. The project provides extensive documentation and learning resources, including a structured curriculum and hands-on projects, to assist with onboarding and architectural mastery. It is distributed as a single binary, facilitating deployment across diverse infrastructure ranging from resource-constrained devices to large-scale distributed cloud clusters.

This project is a distributed, document-oriented database system designed to store information in flexible, hierarchical structures. It supports horizontal scaling through automated sharding and maintains high availability across global clusters using a multi-node replication protocol. By executing multi-document operations as atomic units, the system ensures data integrity and consistency across distributed environments. The platform distinguishes itself by integrating advanced vector-based indexing, which enables semantic similarity searches alongside traditional geospatial and lexical queries. It functions as an enterprise-grade data platform, incorporating granular access controls, encryption, and auditing mechanisms to meet the requirements of regulated production environments. These capabilities allow for the management of large-scale datasets while maintaining the flexibility of a schema-less storage model. The system provides a comprehensive suite of tools for database administration, including command-line utilities for infrastructure management, data migration, and performance monitoring. It supports integration with container orchestration platforms and offers standardized client libraries to facilitate connectivity across various programming languages and business intelligence tools.

FlatBuffers is a cross-platform serialization library designed for performance-critical applications that require efficient, zero-copy data access. By organizing data in a structured binary format, it allows applications to read and write complex data structures directly from memory-mapped buffers without the need for intermediate parsing or temporary object allocation. The project distinguishes itself through a schema-driven approach that balances high-performance access with long-term data evolution. It utilizes a unique memory layout featuring relative offsets and inline fixed-size structures, which ensures that data remains position-independent and cache-friendly. To maintain compatibility as data requirements change, the system employs a table-based format that allows for the addition or deprecation of fields without breaking existing readers. Furthermore, it supports polymorphic data through union-type tagged offsets, providing a flexible yet strictly typed mechanism for handling varied data structures. Beyond its core serialization capabilities, the project provides a comprehensive ecosystem for cross-language interoperability. It includes a schema compiler that generates native source code for a wide range of programming languages, enabling type-safe access to binary data across disparate systems. The library also offers dynamic, schema-less serialization options for scenarios requiring greater flexibility, alongside robust tooling for data format conversion and build system integration. The project is distributed with extensive support for major build systems and package managers, facilitating integration into diverse development environments. Its documentation and tooling support the entire lifecycle of data contract development, from initial schema definition to cross-platform deployment.

CapnProto is a zero-copy serialization framework and remote procedure call system. It serves as a C++ communication library providing a schema-based data interchange format that eliminates the need to encode or decode data before reading it from memory. The system enables high-performance data serialization and low-latency network communication. It supports cross-language data exchange by using a defined schema to ensure consistent binary representation across different platforms. The framework provides tools for implementing remote procedure calls, allowing functions to be invoked on a remote server as if they were local. It handles the serialization of complex data structures for efficient transmission and storage across distributed systems.