cube2222/octosql
Octosql
Octosql 是一个联邦 SQL 查询引擎、数据转换器和流式 SQL 处理器。它允许用户跨多个异构数据源（包括不同类型的数据库和文件格式）执行单一 SQL 语句，从而合并并转换结果集。
该系统的独特之处在于将 CSV、JSONLines 和 Parquet 文件视为虚拟表，并利用基于插件的架构扩展对外部存储引擎的连接。它作为无限数据流的流式处理器，使用水印（watermarks）、撤回（retractions）和翻滚窗口（tumbling windows）来维持乱序事件的一致性。此外，它还可用作 SQL 数据生成器，通过表值函数生成合成数据集和记录流。
该引擎具备跨源数据连接和多源分析能力，并通过源端谓词下推（predicate push-down）进行优化，以减少数据传输。它通过包含联合类型的静态类型系统管理复杂数据，并提供查询执行计划可视化功能以增强可观测性。
Features
- Cross-Source Data Integration - Joins and transforms data from multiple different database engines and file formats using a single SQL interface.
- Federated SQL Query Tools - Provides a federated SQL interface to execute single statements across multiple disparate data sources and merge results.
- Cross-Source Joins - Executes queries that combine and transform information originating from different database engines or file formats into a single result set.
- Virtual Table Transformations - Treats CSV, JSONLines, and Parquet files as virtual tables for analysis and transformation via SQL.
- Streaming Processors - Processes infinite data streams using watermarks and windowing to maintain consistency for out-of-order events.
- Multi-Source Queries - Runs single SQL statements across diverse databases to merge data from different origins into a unified view.
- Database Connectivity Plugins - Deno Query allows integrating third-party plugins to query information across diverse database types and external storage systems.
- Database Connector Plugins - Provides a plugin system to add native connectivity for additional databases like PostgreSQL and MySQL.
- Infinite Stream Executions - Executes queries on endless data streams using watermarks and retractions to handle out-of-order events.
- Real-Time Data Streaming - Processes infinite data streams using SQL windows and watermarks to maintain consistency for out-of-order events.
- SQL Query Execution Engines - Implements a query engine that processes standard SQL commands to join and transform data across different database types and file formats.
- File Format Querying - Processes JSON, CSV, TSV, and Parquet files by treating them as virtual tables for SQL queries.
- Multi-Dataset Analysis - Analyzes combined datasets from relational tables and various file formats without migrating data to a central location.
- Virtual Tables - Treats files like CSV and Parquet as virtual tables to allow SQL querying without data ingestion.
- Watermark-Based Event Tracking - Uses watermarks to coordinate out-of-order event streams and trigger the completion of time-based windows.
- Plugin-Based Architectures - Uses a plugin architecture to extend connectivity to various external storage engines and database types.
- Synthetic Data Generators - Produces synthetic datasets and record streams using table-valued functions for simulation and testing.
- Stream Joining - Implements in-memory joining of live data streams to merge records based on shared keys.
- Tumbling Windows - Organizes streaming records into tumbling windows and triggers output updates based on record counts or watermarks.
- Predicate Pushdown Joins - Minimizes data transfer by matching small input records against large external sources using source-side predicate filtering.
- Predicate Pushdown - Optimizes performance by pushing filters directly to the data source to reduce record transfer volume.
- Query Performance Analyzers - Visualizes query execution plans and utilizes predicate push-down to optimize joins against massive remote tables.
- Retraction-Based State Updates - Ensures consistency in infinite streams by issuing retractions to correct previously emitted out-of-order results.
- Standard Input Virtual Tables - Pipes information from standard input into the engine, treating live streams as virtual tables.
- Stream Join Operators - Combines two live data streams in memory using watermarks to buffer records and ensure consistent results.
- Database Tools - Multi-database SQL query tool.
- Query Engines - Joins and transforms data across multiple databases and formats.
