erikgrinaker/toydb
Toydb
ToyDB is a distributed SQL database that provides a system for storing and querying data across multiple nodes. It focuses on maintaining strong consistency and fault tolerance through the implementation of a distributed consensus algorithm.
The project distinguishes itself by supporting historical data versioning, enabling time-travel queries to retrieve the state of the database from a specific point in the past. It utilizes multi-version concurrency control to manage ACID transactions and ensure data integrity during concurrent operations.
The system covers relational data modeling with table schemas, primary keys, and foreign keys to enforce referential integrity. Its query engine parses SQL statements and executes them through an optimized pipeline of operators, supporting joins, aggregations, and heuristic query optimization. Data is persisted via pluggable storage backends, including log-structured and in-memory engines.
Features
- Distributed SQL Databases - Implements a relational database engine designed to scale horizontally while maintaining transactional consistency.
- Raft Consensus Implementations - Synchronizes a command log across a cluster of nodes using the Raft consensus algorithm for fault tolerance.
- ACID Transactional Cores - Ensures data integrity using snapshot isolation and multi-version concurrency control to prevent dirty and phantom reads.
- Historical Data Querying Interfaces - Allows retrieval of the database state from specific points in the past using logical timestamps.
- Time-Travel Query Interfaces - Enables retrieval of past table states using snapshot identifiers or transaction timestamps.
- Data Joins - Implements general purpose joining functionality to merge columns from multiple tables based on shared keys.
- Data Querying - Provides interfaces for filtering, sorting, and retrieving stored data collections.
- Data Replication - Synchronizes an ordered command log across distributed nodes to ensure data consistency and availability.
- Table Schemas - Supports the creation of tables with specific data types, primary keys, unique constraints, and foreign key relationships.
- SQL Query Execution - Executes structured SQL statements and transactions using an optimized engine to retrieve or modify data.
- Schema Definitions - Provides capabilities to create and remove tables with basic datatypes, primary keys, foreign keys, and column indexes.
- Distributed Consensus Protocols - Replicates state across a cluster using a consensus protocol to maintain consistency across all nodes.
- Consensus Algorithms - Implements consensus algorithms to ensure all nodes in the cluster agree on the sequence of operations.
- Fault Tolerant Storage - Maintains data availability and operational status as long as a majority of cluster nodes remain functional.
- Multi-Version Concurrency Control - Tracks data versions via logical timestamps to provide snapshot isolation and enable time-travel queries.
- Multi-Version Concurrency Controls - Uses multi-version concurrency control to enable lock-free reads and writes via data versioning.
- Multi-Version Concurrency Control - Utilizes multi-version concurrency control to provide snapshot isolation and ensure data integrity during concurrent operations.
- Query Execution Pipelines - Processes SQL statements by pulling and transforming data rows through a pipeline of operator nodes.
- Referential Integrity Constraints - Blocks deletions, updates, or insertions that violate primary key constraints or foreign key references.
- Relational Data Modeling - Provides capabilities for organizing data into structured tables with primary keys and foreign key relationships.
- Row Deletions - Removes rows from a table that match a specific boolean condition.
- Row Insertions - Provides the ability to add new records to tables using standard SQL insert operations.
- SQL-Based Row Updates - Allows modification of existing records in a table using SQL expressions and conditional predicates.
- SQL Query Execution Engines - Provides an engine that processes standard SQL commands while maintaining strict consistency and isolation.
- Strong Consistency Guarantees - Guarantees linearizability by routing writes through a leader and confirming them via a quorum.
- Table Deletion - Permanently deletes a table and all its associated data from the system.
- Log-Structured Storage Engines - Persists data to an append-only file and maintains an in-memory index of keys to file offsets.
- Replicated State Machines - Maintains identical state across all nodes by applying an ordered sequence of transactions.
- Write Conflict Resolution Policies - Detects concurrent transactions modifying the same key and resolves conflicts using specific policies.
- Mathematical Evaluators - Computes results using standard arithmetic operators, floating point math, and three-valued logic.
- Grouped Aggregations - Reduces grouped data into scalar summaries such as average, count, and sum.
- Append-Only Log Stores - Writes data to an append-only log while maintaining an in-memory map of keys to file offsets.
- Pluggable Storage Drivers - Decouples database logic from the persistence layer using a pluggable interface for interchangeable storage engines.
- Pluggable Storage Engines - Supports interchangeable storage backends, including log-structured and in-memory engines.
- Query Planning - Rewrites logical execution plans using heuristic strategies to reduce data transmission and computation.
- SQL Abstract Syntax Tree Parsing - Processes SQL queries into abstract syntax trees to enable programmatic analysis and execution.
- SQL Aggregate Functions - Provides standard SQL aggregate functions for calculating sums, counts, and averages over datasets.
- Storage Backend Adapters - Supports swapping between different data persistence layers such as log-structured stores or in-memory engines.
- SQL Expression Evaluation - Computes results using logical, comparison, mathematical, and string operators including pattern matching with wildcards.
- Storage Abstractions - Uses trait-based interfaces to decouple the core database logic from the underlying persistence layer.
- Database Systems - Distributed SQL database implementation for educational purposes.
- Databases & Data - Distributed SQL database implementation for educational purposes.
