mit-pdos/noria
Noria
Noria este un motor de dataflow dinamic și o bază de date cu vederi materializate (materialized views) care transformă operațiunile de citire costisitoare în căutări rapide pe bază de cheie. Acesta funcționează ca un sistem cu stare care actualizează incremental vederile dependente pe măsură ce tabelele relaționale de bază se modifică.
Sistemul poate funcționa ca un emulator de protocol MySQL, mimând protocolul binar pentru a permite aplicațiilor să preia date precalculate fără a modifica codul clientului. Poate fi utilizat și ca bază de date embedded în Rust, integrând serverul direct într-un program prin binding-uri native pentru a efectua operațiuni de dataflow fără coordonare externă.
Proiectul acoperă coordonarea distribuită a fluxului de date, inclusiv monitorizarea stării nodurilor de lucru prin heartbeats. Oferă instrumente pentru gestionarea tabelelor relaționale, sincronizarea incrementală a vederilor și gestionarea grafurilor de flux de date.
Observabilitatea este susținută prin colectarea de statistici de domeniu, benchmarking de performanță și un inspector web de date pentru vizualizarea rezultatelor precalculate.
Features
- Dataflow Engines - Implements a stateful dataflow engine that incrementally updates materialized views based on relational table changes.
- State Management - Organizes data as a graph of dependencies where updates flow from source tables to derived views.
- Embedded Database Integration - Allows integrating the server directly into a process to perform data-flow operations without external coordination.
- Key Lookup Operations - Transforms complex relational queries into high-speed point lookups against precomputed materialized views.
- Materialized Views - Creates materialized views from relational queries to provide fast data retrieval by calculating results in advance.
- Incremental View Refreshes - Automatically updates precomputed query results by propagating changes from base tables through a dependency graph.
- Rust Database Bindings - Provides native Rust bindings for interacting with the server or embedding the server directly into a Rust application.
- Embedded Database Engines - Offers a high-performance data storage server that can be embedded directly into Rust programs via native bindings.
- Dataflow Frameworks - Integrates a stateful dataflow server directly into a Rust application to perform high-speed operations.
- Native Server Embeddings - Provides a Rust-based interface for direct server embedding to eliminate protocol overhead.
- Coordinator-Worker Topologies - Implements a coordinator-worker architecture to monitor worker health via heartbeats and distribute read requests.
- Database Wire Protocol Emulations - Implements the MySQL wire protocol to support standard drivers and legacy client libraries without requiring code changes.
- High-Performance Databases - Reduces latency in web applications by transforming expensive database queries into fast keyed lookups.
- MySQL Compatibility - Provides a translation layer that converts standard MySQL queries into system requests for retrieving precomputed data.
- MySQL Protocol-Based Migrations - Enables moving applications from traditional databases to a dataflow system using the MySQL protocol to avoid changing client code.
- MySQL Wire Protocol Servers - Emulates the MySQL binary wire protocol to support legacy client libraries without requiring code changes.
- Relational Data Storage - Supports writing records into base relational tables to trigger automatic updates for dependent materialized views.
- Dataflow Graph Management - Provides tools for tracking data flow graph structures and performing controlled migrations to maintain consistency.
- Distributed Coordination Systems - Manages worker nodes and data domains to synchronize state and maintain consistency across a distributed system.
- Database Systems - Stateful data-flow engine for web application backends.
- Databases & Data - Stateful data-flow engine for web application backends.
