harelba/q
Q
q is a command-line utility for the processing, filtering, and aggregation of tabular text and database files using standard SQL syntax. It functions as a query engine that treats CSV and TSV files, as well as standard input, as relational database tables.
The tool distinguishes itself by providing a persistent cache layer that stores processed tabular data in a binary format to accelerate repeated queries on large datasets. It also maps individual filenames or stream identifiers to relational table names, enabling SQL joins across disparate text files.
The project covers a broad range of data analysis capabilities, including automated schema detection for column types, tabular output formatting, and the ability to export processed in-memory datasets into physical SQLite database files. It integrates directly into Unix pipelines by accepting tabular data via standard input.
Features
- Data Pipeline Tools - Functions as a command-line tool that orchestrates the processing of tabular data from pipes to various export formats.
- SQL Query Interfaces - Enables executing standard SQL queries against delimited text files by treating filenames or standard input as database tables.
- Automatic Type Detection - Identifies data types for each column automatically to ensure numeric comparisons and aggregations function correctly.
- Data Joins - Combines data from different files using join operations to link datasets based on shared columns.
- Standard Input Ingestion - Integrates with the shell by treating the stdin stream as a queryable table within the SQL execution context.
- Shell Integration - Integrates SQL querying directly into Unix shell workflows by accepting tabular data from standard input.
- SQL Engines - Acts as a relational query processor that treats CSV and TSV files as tables for SQL execution.
- In-Memory Virtualization - Treats text files and standard input as temporary relational tables by loading them into a volatile SQLite instance.
- Schema Inference - Analyzes input files to determine column names from headers and infer data types for every field.
- Relational Transformations - Provides the ability to join and reshape delimited text files using standard SQL logic for reports and processing.
- Tabular Data Processors - Implements a utility for joining, filtering, and aggregating delimited text data using standard SQL syntax.
- SQL Command Line Interfaces - Runs SQL queries on CSV or TSV files directly from the terminal to extract insights without a database server.
- Virtual Table Mappings - Maps filenames and stream identifiers to relational table names to enable SQL joins across disparate files.
- Tabular Data Type Inference - Automatically assigns SQL data types to tabular columns by analyzing initial rows.
- Persistent Binary Caches - Stores processed tabular data in a persistent binary format to accelerate repeated queries on large datasets.
- Tabular Data Caching - Stores processed tabular data in a binary format to bypass repetitive parsing of large raw text files.
- SQLite Integration - Provides direct support for running SQL queries against physical SQLite database files.
- Tabular Data Caching - Converts raw text files into persistent database formats to speed up repeated queries on large volumes of data.
- Command Line Utilities - Executes SQL queries on CSV and TSV files.
- Text-to-SQLite - Listed in the “Text-to-SQLite” section of the Awesome Sqlite awesome list.
