multiprocessio/dsq
Dsq
dsq is a command-line interface and data engine for executing SQL queries against local structured files, such as CSV, JSON, Parquet, and Excel, without requiring a formal database import. It functions as a schema-inference engine that automatically detects data types and maps heterogeneous file structures into relational tables for analysis.
The tool utilizes a lazy stream data processor and checksum-based disk caching to handle large datasets with minimal memory usage. It provides a persistent interactive shell for iterative data exploration, allowing users to inspect inferred schemas and run sequential SQL statements in real time.
The system supports cross-format joins, nested data querying, and regular expression filtering. It also includes utilities for transforming structured data into JSON format and rendering query results as formatted tables.
Features
- SQL Query Execution - Enables the execution of SQL commands against structured local files like JSON or CSV without a formal database import.
- Schema Inference - Provides an engine that automatically detects column names and data types from raw files to build relational schemas.
- Flat-File SQL Analysis - Executes SQL queries against CSV, JSON, and Parquet files without importing them into a formal database engine.
- Heterogeneous Data Joining - Joins and combines data from different file formats like Excel and JSON using relational database operations.
- Heterogeneous Format Mapping - Translates various file types like CSV, JSON, and Parquet into a unified relational format for SQL joins.
- Large Dataset Streaming - Analyzes large files with minimal memory usage through lazy loading and disk-backed caching.
- Lazy Stream Processing - Processes records one by one from source files to minimize memory usage during large dataset operations.
- Interactive SQL Executors - Provides a persistent shell for iterative data exploration and schema inspection using standard SQL on local files.
- File Format Querying - Combines data from different file formats by treating each individual file as a separate table in a SQL query.
- Virtual Tables - Maps external file structures to relational database tables without requiring a formal data import process.
- Database REPL Shells - Provides an interactive shell with a persistent session for running sequential SQL statements.
- Ad Hoc Dataset Querying - Provides an interactive shell to inspect schemas and iteratively query local datasets for quick insights.
- Checksum-Based Disk Caching - Implements disk-backed caching using file checksums to accelerate subsequent queries on static datasets.
- Checksum-Based Disk Caches - Uses checksum-based disk caching to accelerate subsequent queries by avoiding redundant processing of static files.
- Data Format Transformations - Transforms files between different formats by applying SQL transformations to the input data.
- JSON Object Querying - Allows access to values within nested objects or spreadsheets by defining a path to the target array or object.
- Schema Inspections - Allows users to inspect and verify the automatically detected data types and structures of their local files.
- Interactive REPLs - Offers an interactive environment for executing multiple sequential SQL statements to analyze datasets in real time.
- String-to-Numeric Conversions - Automatically converts string values in delimited files to integers or floats for accurate sorting and computation.
- Query Engines - Runs SQL queries against JSON, CSV, Excel, and Parquet.
