Open-source alternatives to PalDB

Chronicle Queue is a high-performance data handling system featuring off-heap message queues, memory-mapped file stores, and replicated message stores. It provides a binary compatible memory layout that enables different programming languages to share data without serialization overhead. The system utilizes a replicated message store to synchronize data across multiple nodes, ensuring high availability and instant failover. Its memory-mapped architecture supports deterministic replay from disk and low-latency data recording. The project implements off-heap memory management and zero-allocati

Kvrocks is a disk-based NoSQL database and distributed key-value store that leverages the RocksDB storage engine to persist large datasets to physical disk. It is designed to be a Redis-compatible database, utilizing the standard Redis communication protocol to ensure interoperability with existing client libraries and tools. The project distinguishes itself by combining a disk-persistent storage model with advanced retrieval capabilities, including vector search for k-nearest neighbor queries, full-text search indexing, and geospatial query execution. It supports distributed clustering with

RoseDB is a persistent key-value database and log-structured storage engine. It functions as a lightweight storage system that utilizes a log-structured hash table and a Bitcask engine implementation to provide fast data retrieval and disk-backed persistence. The system operates as an atomic transaction engine, grouping multiple read and write operations into single units to maintain data consistency. It handles data through a key-value model that supports individual insertions, lookups, and deletions. The database provides capabilities for batch data processing and atomic updates. Additiona

Level is a database library that provides a unified interface for managing sorted key-value data. It functions as an abstraction layer that allows applications to store and retrieve binary information consistently across server-side environments and web browsers. The project utilizes a modular architecture that supports pluggable storage backends, enabling the system to adapt to different host environments while maintaining identical behavior. By organizing data in lexicographical order, it facilitates efficient range queries and ordered retrieval. The library handles large datasets through a

goleveldb is an embedded key-value storage database for Go. It provides local data persistence and indexing, allowing applications to store and retrieve information using unique keys without requiring a separate server. The database organizes data using a log-structured merge-tree and persistent indexing in lexicographical order. This structure supports efficient range scans and prefix-based searches. The system includes capabilities for atomic batch writes to ensure data consistency and avoid partial updates. Performance is managed through write-ahead logging, memory-based buffering, and bl

BuntDB is an embedded key-value store for Go applications, providing in-memory storage with optional disk persistence. It structures data using a B-tree for ordered key-value access and an R-tree for spatial indexing, allowing both range scans and geometric intersection queries. Support for indexing on nested JSON document fields enables efficient lookups by values within JSON objects, and per-key time-to-live (TTL) expiration automatically removes stale entries. The store uses copy-on-write transaction isolation, ensuring each transaction sees a consistent snapshot and changes are applied at

MMKV is a high-performance, cross-platform key-value storage framework designed for mobile platforms and POSIX environments, including Android, iOS, macOS, and Windows. It provides a persistence layer that utilizes memory-mapped files and binary serialization to achieve low-latency data access. The project distinguishes itself through native support for multi-process synchronization, allowing concurrent read and write operations across different application processes. It also implements security via AES encryption for data at rest, featuring symmetric encryption and key rotation to protect st

Pebble is an embedded key-value storage engine written in Go, designed as a library that provides durable, write-optimized data persistence directly within applications. It organizes data using a log-structured merge-tree (LSM-tree) structure, where writes are first buffered in an in-memory skiplist memtable and persisted to a write-ahead log before being flushed to block-based SSTable files on disk. The engine supports atomic batch commits, configurable write synchronization, and automatic background compaction that merges and rewrites sorted runs to reclaim space and maintain read performanc

Kvrocks is a distributed key-value store and Redis-compatible NoSQL database. It utilizes a RocksDB storage engine to provide disk-based persistence, allowing for high-capacity data storage with reduced memory costs compared to in-memory systems. The system functions as a vector database and full-text search engine, supporting nearest-neighbor searches on vector embeddings and complex document queries via text matching. It employs a proxyless cluster architecture with slot-based routing to distribute data and scale capacity across multiple nodes. The platform covers a wide range of data mana

Mini-LSM is an educational storage engine and key-value database library designed to demonstrate the implementation of log-structured merge-tree architecture. It serves as a pedagogical resource for understanding how to build high-performance storage systems from the ground up, focusing on the mechanics of persistent data structures and disk-based storage. The project provides a functional framework for managing data through memory-to-disk flushing and multi-version concurrency control. It distinguishes itself by implementing snapshot-based isolation, which allows for consistent views of the

LMDB is an embedded key-value storage engine that provides ACID-compliant data persistence. It is a memory-mapped database that utilizes B+ trees to store key-value pairs, ensuring atomicity, consistency, isolation, and durability. The engine maps files directly into the virtual address space to minimize data copying and system calls. This approach enables high-performance local caching and low-latency data access, specifically optimizing for read-heavy database workflows. The system implements a transactional model with copy-on-write versioning and single-writer multi-reader locking. These

RoseDB is an embedded key-value database and log-structured storage engine. It functions as a library-based database that runs within an application process to provide local data persistence without the need for a separate database server. The project implements a Bitcask-style store, utilizing an append-only log format to maximize write throughput and minimize seek time. It ensures data consistency by grouping multiple read and write operations into single atomic units. The storage architecture relies on an in-memory key index for constant-time lookups and uses segment-based file management

Redka is a SQL-backed key-value store that implements a Redis-compatible interface. It acts as a relational data bridge, translating non-relational protocol commands into relational queries to provide persistence and ACID transactions using a SQL database backend. The system allows for relational data introspection by exposing internal key-value data as SQL views, enabling direct querying and analysis via relational syntax. It can be deployed as a standalone network server or as an embedded library within a Go application. The project supports a variety of data structures, including strings,

HexFiend is a native macOS hex editor designed for opening and editing binary files of any size without loading them entirely into memory. It uses a 64-bit addressable data model with memory-mapped file access, enabling navigation of multi-gigabyte files while maintaining byte-level undo for insert and delete operations. The editor distinguishes itself through a diff algorithm for binary file comparison that accounts for insertions and deletions, and a template-based structure parsing system that uses user-defined scripts to visualize binary file layouts. It also includes a typed data interpr

This project provides a high-performance key-value storage solution for mobile applications, utilizing memory-mapped files to enable rapid read and write operations. It functions as a persistent data layer that integrates directly with the JavaScript runtime to minimize overhead, while supporting reactive state synchronization to ensure interface components update automatically when stored data changes. The storage system distinguishes itself through its ability to manage multiple independent database instances, which organizes data by module and prevents collisions between application compon

rkyv is a zero-copy deserialization framework for Rust that provides a binary serialization format for memory-mappable data archives. It allows complex data structures to be mapped to bytes and accessed directly from a buffer without allocating new memory or copying data. The project enables the serialization of polymorphic types and trait objects, maintaining their dynamic behavior and structure within the binary form. It utilizes relative-pointer addressing and byte-aligned structure packing to ensure data remains valid regardless of where it is loaded in memory. The framework covers high-

bbolt is an ACID-compliant embedded key-value store for Go applications. It persists all data in a single memory-mapped file on disk, organizing information using B+ trees to facilitate sorted key iteration and efficient range queries. The project distinguishes itself through a hierarchical data organization model, allowing buckets to be nested within other buckets to create a tree-like structure. It employs a single-writer, multi-reader locking mechanism and copy-on-write transactions to ensure serializable isolation and data integrity. The system includes comprehensive data management capa

Bolt is a single-file embedded key-value store for Go applications. It is an ACID transactional database that organizes data in B+trees on disk to provide efficient sorted key retrieval and range scans. The system uses a memory-mapped model to map the database file directly into the process address space for fast random-access reads. The project distinguishes itself through a multi-version concurrency control architecture that allows multiple simultaneous readers to access a consistent snapshot of data without blocking a writer. It employs a single-writer multi-reader locking model and uses a

PikiwiDB is a distributed NoSQL database and disk-based key-value store that serves as a Redis-compatible protocol server. It is designed to handle datasets larger than available system memory by utilizing a persistence engine that stores the full dataset on disk. The system employs a tiered storage model, caching frequently accessed hot data in memory while maintaining the primary volume on disk. It ensures high availability through a replicated data store architecture, using asynchronous binary logs to synchronize data between primary and secondary nodes. The project supports distributed d

The 1BRC (One Billion Row Challenge) is a Java performance benchmarking exercise that processes one billion temperature records from a text file to compute the minimum, mean, and maximum temperature per weather station. At its core, it is a large-scale data aggregation challenge designed to test how efficiently a Java program can parse and aggregate structured data from a plain text file, serving as both a programming exercise and a benchmark for Java performance optimization. The project distinguishes itself through a collection of performance-oriented architectural patterns for high-through

Vaex is a high-performance Apache Arrow DataFrame library and out-of-core data processing engine designed to handle billion-row tabular datasets in Python. It functions as a lazy evaluation framework that defers computations and transformations until results are required, enabling the processing of datasets that exceed available system RAM by mapping files directly from disk. The project distinguishes itself as a tool for big data visualization and exploration, specifically integrated for use within interactive notebooks. It provides specialized capabilities for machine learning feature engin

A fast and clever hex editor for macOS

Gson is a Java library designed for the serialization and deserialization of objects into structured text formats. It functions as a reflection-based data mapper, inspecting class structures at runtime to automatically convert memory-resident objects into data representations and reconstruct them back into typed language objects. The library distinguishes itself through a modular type-adapter pattern that allows for custom conversion rules for complex or nested structures. It also provides robust support for production environments by generating build-time metadata and configuration rules, wh

This project is a Protocol Buffers Go compiler and code generation framework that translates schema definitions into optimized Go structures and reflection-free serialization methods. It includes a gRPC service generator for producing client and server communication code and a serialization performance toolkit for generating automated benchmarks and test suites. The framework features a plugin-based system for transforming data definitions into source code with custom field mapping and struct tag injection. It allows for custom type mapping and struct type overriding to link data fields to sp

Canopy is the official Go implementation of a blockchain node that runs a recursive network architecture, enabling new blockchains to launch as dependent layers that can later graduate into fully independent security roots. The project provides a hybrid consensus mechanism that combines Byzantine fault-tolerant Proof-of-Stake with Verifiable-Delay Functions, delivering instant finality while protecting against long-range attacks through trustless chain age verification. Validators secure multiple chains simultaneously through restaking, where a single bonded stake serves as collateral across t

Fory is a cross-language serialization framework and binary data serializer designed to convert complex object graphs into a compact binary format for high-performance data exchange. It includes an IDL-based schema compiler to transform interface definition language files into type-safe native data models and a schema evolution manager to maintain forward and backward compatibility. The project features a zero-copy data access layer that allows reading specific fields from binary rows without deserializing the entire object. It supports dual-mode serialization, enabling a toggle between a por