cube2222/octosql
Octosql
Octosql es un motor de consultas SQL federado, transformador de datos y procesador de SQL en streaming. Permite a los usuarios ejecutar sentencias SQL únicas a través de múltiples fuentes de datos dispares, incluyendo diferentes tipos de bases de datos y formatos de archivo, para combinar y transformar resultados en un conjunto unificado.
El sistema se distingue por tratar archivos CSV, JSONLines y Parquet como tablas virtuales y utilizar una arquitectura basada en plugins para extender la conectividad a motores de almacenamiento externos. Funciona como un procesador de streaming para flujos de datos infinitos, utilizando marcas de agua (watermarks), retracciones y ventanas deslizantes (tumbling windows) para mantener la consistencia en eventos fuera de orden. Además, sirve como generador de datos SQL capaz de producir conjuntos de datos sintéticos y flujos de registros mediante funciones con valores de tabla.
El motor incluye capacidades para realizar joins entre fuentes de datos y análisis multi-fuente, optimizado mediante el push-down de predicados en el lado de la fuente para reducir la transferencia de datos. Gestiona datos complejos a través de un sistema de tipos estáticos con tipos unión y proporciona observabilidad mediante la visualización de planes de ejecución de consultas.
Features
- Cross-Source Data Integration - Joins and transforms data from multiple different database engines and file formats using a single SQL interface.
- Federated SQL Query Tools - Provides a federated SQL interface to execute single statements across multiple disparate data sources and merge results.
- Cross-Source Joins - Executes queries that combine and transform information originating from different database engines or file formats into a single result set.
- Virtual Table Transformations - Treats CSV, JSONLines, and Parquet files as virtual tables for analysis and transformation via SQL.
- Streaming Processors - Processes infinite data streams using watermarks and windowing to maintain consistency for out-of-order events.
- Multi-Source Queries - Runs single SQL statements across diverse databases to merge data from different origins into a unified view.
- Database Connectivity Plugins - Deno Query allows integrating third-party plugins to query information across diverse database types and external storage systems.
- Database Connector Plugins - Provides a plugin system to add native connectivity for additional databases like PostgreSQL and MySQL.
- Infinite Stream Executions - Executes queries on endless data streams using watermarks and retractions to handle out-of-order events.
- Real-Time Data Streaming - Processes infinite data streams using SQL windows and watermarks to maintain consistency for out-of-order events.
- SQL Query Execution Engines - Implements a query engine that processes standard SQL commands to join and transform data across different database types and file formats.
- File Format Querying - Processes JSON, CSV, TSV, and Parquet files by treating them as virtual tables for SQL queries.
- Multi-Dataset Analysis - Analyzes combined datasets from relational tables and various file formats without migrating data to a central location.
- Virtual Tables - Treats files like CSV and Parquet as virtual tables to allow SQL querying without data ingestion.
- Watermark-Based Event Tracking - Uses watermarks to coordinate out-of-order event streams and trigger the completion of time-based windows.
- Plugin-Based Architectures - Uses a plugin architecture to extend connectivity to various external storage engines and database types.
- Synthetic Data Generators - Produces synthetic datasets and record streams using table-valued functions for simulation and testing.
- Stream Joining - Implements in-memory joining of live data streams to merge records based on shared keys.
- Tumbling Windows - Organizes streaming records into tumbling windows and triggers output updates based on record counts or watermarks.
- Predicate Pushdown Joins - Minimizes data transfer by matching small input records against large external sources using source-side predicate filtering.
- Predicate Pushdown - Optimizes performance by pushing filters directly to the data source to reduce record transfer volume.
- Query Performance Analyzers - Visualizes query execution plans and utilizes predicate push-down to optimize joins against massive remote tables.
- Retraction-Based State Updates - Ensures consistency in infinite streams by issuing retractions to correct previously emitted out-of-order results.
- Standard Input Virtual Tables - Pipes information from standard input into the engine, treating live streams as virtual tables.
- Stream Join Operators - Combines two live data streams in memory using watermarks to buffer records and ensure consistent results.
- Database Tools - Multi-database SQL query tool.
- Query Engines - Joins and transforms data across multiple databases and formats.
