PRQL/prql
Prql
PRQL is a functional, modular data transformation language that serves as a compiler for relational data pipelines. It allows developers to write expressive, pipelined queries that are translated into standard SQL dialects. By abstracting complex data manipulation into a readable, sequential syntax, the project enables the construction of maintainable workflows that remain independent of specific database engines.
The language distinguishes itself through a robust compilation infrastructure that performs type validation and relational algebra analysis before generating target-specific code. It supports modular namespace resolution and reusable function definitions, allowing for the creation of complex, hierarchical data projects. Developers can integrate these transformations directly into various programming environments or notebook interfaces, while maintaining the ability to embed raw SQL for specialized database features.
The project provides a comprehensive suite of data manipulation primitives, including support for windowed transformations, conditional logic, and complex aggregations. It also includes diagnostic tools for tracking column lineage and visualizing query transformation flows. The command-line interface facilitates project automation, dependency management, and real-time query previews, while editor-based syntax highlighting and grammar definitions support development productivity.
Features
- Transpilers - Compiles high-level, functional data transformation pipelines into standard SQL for execution across multiple database engines.
- Data Transformation Pipelines - Translates a declarative, pipelined syntax into standard database query languages to simplify complex data manipulation logic.
- Transformation Chains - Processes data by chaining sequential operations where the output of each step flows directly into the next transformation.
- Data Transformation Languages - Provides a functional, modular language designed to simplify complex data manipulation through sequential pipeline operations.
- Database Query Translators - Translates expressive query syntax into native SQL dialects while supporting raw SQL embedding.
- Data Processing Pipelines - Provides a framework for building maintainable data processing workflows that abstract away verbose SQL syntax.
- Query Translators - Translates high-level data transformation pipelines into native SQL syntax for efficient execution.
- SQL Query Builders - Translates functional query syntax into optimized database-specific SQL to simplify complex manipulation logic.
- SQL Dialect Adapters - Maps internal logical representations to specific SQL flavors by applying engine-specific syntax rules and keyword escaping.
- Function Definitions - Creates reusable logic blocks using a functional syntax that accepts parameters and returns computed values.
- Data Operation Pipelines - Enables chaining multiple data processing operations into sequential, readable workflows.
- Analytical Data Engines - Manages complex data transformations and aggregations through reusable functions and hierarchical modules.
- Data Aggregation Tools - Computes summary statistics across rows by distilling multiple values into single results using functions like sum, count, or average.
- Interactive Data Exploration Tools - Enables live feedback and rapid iteration during data exploration within notebooks and editors.
- Query Previewers - Displays generated SQL code in a side-by-side pane while editing queries to provide immediate feedback on transformation logic.
- Intermediate Representations - Uses a formal logical model to represent data transformations, enabling optimization and analysis before final query emission.
- Column-Level Lineage Extraction - Generates graphs representing the flow and transformation of data columns to track dependencies and data provenance.
- Column Transformation - Adds calculated columns to a relation using constants or expressions based on existing column values.
- Relational Join Engines - Combines rows from multiple relations based on matching criteria using standard join types.
- Row-wise Transformations - Applies pipelines to segments of rows defined by relative positions or value ranges.
- Notebook Execution Environments - Enables interactive data transformation scripts within notebook environments for immediate analysis.
- CLI Productivity Tools - Provides a command-line interface to resolve dependencies, execute scripts, format code, and generate shell completions.
- Arithmetic Operations - Executes standard arithmetic, trigonometric, and logarithmic operations on data fields to support complex calculations.
- Conditional Branching - Executes branching operations using case statements to map input values to specific outputs based on defined criteria.
- Transformation Validators - Enforces schema constraints and data structure integrity during the transformation process before generating executable code.
- Data Import - Reads data from common file formats like Parquet, CSV, and JSON to integrate local datasets into transformation pipelines.
- Conditional Row Filters - Restricts rows in a relation by applying conditional logic that evaluates to true or false.
- Query Parameterization - Supports inserting placeholders into queries to ensure dynamic and secure execution.
- Query Sorting - Orders query results by specifying columns and directions using a concise syntax for ascending or descending sequences.
- Raw SQL Execution - Allows embedding raw database-specific SQL statements directly into the query pipeline.
- Configuration Extensions - Allows users to annotate source code, define reusable transformations, organize modules, and specify language versions.
- Data Structure and Type Helpers - Enforces type constraints on variables and function parameters to ensure data integrity during compilation.
- Data Structures - Groups multiple fields or items into structured containers and homogeneous collections to facilitate batch processing.
- Abstract Syntax Tree Tools - Translates high-level source code into intermediate tree structures for analysis and transformation before final query generation.
- Namespace Scopes - Organizes code into hierarchical scopes to manage dependencies and resolve variable references across multiple project files.
