mit-pdos/noria
Noria
Noria is a dynamic dataflow engine and materialized view database that transforms expensive read operations into fast keyed lookups. It operates as a stateful system that incrementally updates dependent views as underlying relational base tables change.
The system can function as a MySQL protocol emulator, mimicking the binary protocol to allow applications to fetch precomputed data without changing client code. It can also be used as a Rust embedded database, integrating the server directly into a program via native bindings to perform dataflow operations without external coordination.
The project covers distributed dataflow coordination, including worker node health tracking via heartbeats. It provides tools for relational table management, incremental view synchronization, and data flow graph management.
Observability is supported through domain statistics collection, performance benchmarking, and a web data inspector for browsing precomputed results.
Features
- Dataflow Engines - Implements a stateful dataflow engine that incrementally updates materialized views based on relational table changes.
- State Management - Organizes data as a graph of dependencies where updates flow from source tables to derived views.
- Embedded Database Integration - Allows integrating the server directly into a process to perform data-flow operations without external coordination.
- Key Lookup Operations - Transforms complex relational queries into high-speed point lookups against precomputed materialized views.
- Materialized Views - Creates materialized views from relational queries to provide fast data retrieval by calculating results in advance.
- Incremental View Refreshes - Automatically updates precomputed query results by propagating changes from base tables through a dependency graph.
- Rust Database Bindings - Provides native Rust bindings for interacting with the server or embedding the server directly into a Rust application.
- Embedded Database Engines - Offers a high-performance data storage server that can be embedded directly into Rust programs via native bindings.
- Dataflow Frameworks - Integrates a stateful dataflow server directly into a Rust application to perform high-speed operations.
- Native Server Embeddings - Provides a Rust-based interface for direct server embedding to eliminate protocol overhead.
- Coordinator-Worker Topologies - Implements a coordinator-worker architecture to monitor worker health via heartbeats and distribute read requests.
- Database Wire Protocol Emulations - Implements the MySQL wire protocol to support standard drivers and legacy client libraries without requiring code changes.
- High-Performance Databases - Reduces latency in web applications by transforming expensive database queries into fast keyed lookups.
- MySQL Compatibility - Provides a translation layer that converts standard MySQL queries into system requests for retrieving precomputed data.
- MySQL Protocol-Based Migrations - Enables moving applications from traditional databases to a dataflow system using the MySQL protocol to avoid changing client code.
- MySQL Wire Protocol Servers - Emulates the MySQL binary wire protocol to support legacy client libraries without requiring code changes.
- Relational Data Storage - Supports writing records into base relational tables to trigger automatic updates for dependent materialized views.
- Dataflow Graph Management - Provides tools for tracking data flow graph structures and performing controlled migrations to maintain consistency.
- Distributed Coordination Systems - Manages worker nodes and data domains to synchronize state and maintain consistency across a distributed system.
- Database Systems - Stateful data-flow engine for web application backends.
- Databases & Data - Stateful data-flow engine for web application backends.
Star history
How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
