Embedded SQL Databases

DuckDB is an embedded, in-process analytical SQL database and OLAP database management system. It functions as a data engine for Parquet and CSV files, allowing users to execute complex SQL queries on large datasets without requiring a separate server process. The system is designed for local analytical processing and embedded data science workflows. It enables the direct querying and analysis of Parquet and CSV files from disk, bypassing the need to load data into a permanent database. The engine provides high-performance analytical SQL execution, including support for window functions and nested subqueries. It incorporates a columnar storage layout and vectorized query execution to handle large-scale data manipulation and exploration. The database is accessible via a standalone command line interface and language-specific bindings for Python, R, Java, and Wasm.

LiteDB is a serverless NoSQL document store and embedded database engine for .NET applications. It persists unstructured documents and binary data into a single standalone disk file, allowing the database to run within the application process rather than as a separate server. The system supports strongly typed queries through Language Integrated Query and allows the execution of standard SQL commands for data retrieval and transformation. It provides native mapping of plain classes into document formats and secures stored information via symmetric-key file encryption. The engine includes capabilities for managing atomic transactions and indexing specific document fields to accelerate search performance. It also supports the storage and streaming of large binary files that exceed standard size limits.

LibSQL is a high-performance, distributed SQL database engine that extends SQLite to support remote network access, edge computing, and real-time synchronization. It functions as an embedded database library that integrates directly into application processes while providing the infrastructure to maintain consistency across multiple geographic regions. The platform distinguishes itself by enabling database interaction over standard HTTP protocols, allowing applications to query remote data sources in serverless and edge environments without requiring local filesystem access. It includes native support for high-dimensional vector similarity search and indexing, enabling AI and machine learning workflows to run directly within the database engine. The system provides a comprehensive suite of tools for managing data lifecycles, including database branching, point-in-time state restoration, and automated synchronization between local replicas and remote primary instances. It also incorporates granular security primitives, such as token-based access control and network-level restrictions, to protect database resources in multi-tenant environments. The project offers extensive observability and administrative features, including query performance monitoring, audit logging, and organizational management tools. It is designed for integration through language-specific drivers and supports advanced data processing through specialized modules for full-text and similarity search.

LiteDB is a serverless, embedded NoSQL document database for .NET applications. It persists data into a single portable file, functioning as a BSON data store that resides within the application process rather than running as a separate server. The system is ACID compliant, utilizing write-ahead logging to ensure atomic, consistent, isolated, and durable transactions. It includes built-in encryption to provide secure local data storage and protect files on disk from unauthorized access. The project covers object-document mapping to convert classes into document formats, indexed search capabilities via B-tree indexing, and specialized streaming for large binary objects. It also provides a dedicated administrative studio for visual data administration and modification.

fmdb is an object-oriented SQLite database library and persistence layer for native macOS and iOS environments. It provides an Objective-C wrapper that encapsulates the low-level C API, allowing applications to manage local relational data storage and embedded database connections through a high-level interface. The library focuses on thread-safe database access by synchronizing operations across multiple threads using serialized queues to prevent data corruption and race conditions. It includes specialized capabilities for secure local storage, such as database encryption and the management of encryption keys. The project covers comprehensive database management, including atomic transaction handling with named save points, schema inspection, and SQL parameter binding to prevent injection. It also provides utilities for result set iteration, dictionary-based row mapping, and the implementation of full-text search using specialized tokenizers.

DuckDB is an in-process analytical database engine designed to run directly within an application process. As a zero-dependency, embedded system, it provides enterprise-grade SQL data processing capabilities without the overhead of managing a dedicated database server. It is built to handle complex analytical and aggregation tasks by storing and retrieving information in columns, allowing for high-performance relational data manipulation. The engine distinguishes itself through a columnar vectorized execution model that maximizes CPU cache efficiency during query operations. It employs adaptive query optimization to dynamically select execution plans at runtime and utilizes zero-copy ingestion to map external data formats directly into memory. To facilitate integration with analytical programming environments, the system supports high-performance data exchange through standardized memory formats and provides specialized connectors for Python, R, and Java. The project covers a broad capability surface, including advanced relational join operations, incremental result streaming for large datasets, and flexible data ingestion from various file formats. It supports complex data types and provides a comprehensive command-line interface for interactive session management and batch processing. The codebase is designed for portability, offering single-file amalgamation to simplify integration into external projects and build systems.

This project is a distributed, document-oriented database system designed to store information in flexible, hierarchical structures. It supports horizontal scaling through automated sharding and maintains high availability across global clusters using a multi-node replication protocol. By executing multi-document operations as atomic units, the system ensures data integrity and consistency across distributed environments. The platform distinguishes itself by integrating advanced vector-based indexing, which enables semantic similarity searches alongside traditional geospatial and lexical queries. It functions as an enterprise-grade data platform, incorporating granular access controls, encryption, and auditing mechanisms to meet the requirements of regulated production environments. These capabilities allow for the management of large-scale datasets while maintaining the flexibility of a schema-less storage model. The system provides a comprehensive suite of tools for database administration, including command-line utilities for infrastructure management, data migration, and performance monitoring. It supports integration with container orchestration platforms and offers standardized client libraries to facilitate connectivity across various programming languages and business intelligence tools.

redb is an embedded key-value store and ACID-compliant storage engine. It functions as a persistent storage system for saving and retrieving data as key-value pairs within a tree structure. The engine is built as an MVCC transactional database, utilizing multi-version concurrency control to manage simultaneous reads and writes without blocking. It employs a single-writer multi-reader model to ensure data consistency while allowing multiple threads to access the store. The system provides persistent state management and atomic transaction management to prevent data corruption during crashes. It handles concurrent data access and ensures that groups of changes are applied as single units.

SurrealDB is a multi-model database engine designed to store and query document, graph, relational, and vector data within a single ACID-compliant platform. It functions as an AI-native data store, integrating vector search, graph traversal, and machine learning model execution directly into its query layer. By providing a unified declarative query language, the platform eliminates the need for external middleware to synchronize data across different storage models. The platform distinguishes itself through its ability to manage agent memory and complex workflows natively. It allows developers to store agent memory, knowledge graphs, and structured data within a single transaction boundary, ensuring consistent state and permissions. Furthermore, the engine supports real-time reactive applications by pushing data updates directly to connected clients through live queries, removing the requirement for external message brokers or polling mechanisms. SurrealDB is built for versatility, operating as a portable database runtime that maintains a consistent interface across embedded, edge, and cloud environments. Its architecture includes a granular, record-level permission model that enforces security and multi-tenant isolation directly at the data layer. The system also features an isolated sandboxing environment for custom extensions, allowing for specialized data processing without compromising system stability or security. The project provides extensive documentation and learning resources, including a structured curriculum and hands-on projects, to assist with onboarding and architectural mastery. It is distributed as a single binary, facilitating deployment across diverse infrastructure ranging from resource-constrained devices to large-scale distributed cloud clusters.