tobymao/sqlglot
Sqlglot
sqlglot is a SQL parser and transpiler that represents queries as abstract syntax trees to enable structural analysis, modification, and semantic transformation. It functions as a dialect translator and query optimizer, converting SQL code between different database engines and simplifying syntax trees through rule-based normalization.
The project provides a framework for defining custom SQL dialects by overriding tokenizers, parsers, and generators. It includes a lineage analyzer to track data flow from source tables through complex queries to identify the origin of specific columns.
Additional capabilities cover programmatic SQL generation, syntax validation, and schema-aware type inference. The system can generate logical plans for query planning, extract metadata, and execute queries in-memory using Python dictionaries as data sources.
Features
- SQL Abstract Syntax Tree Parsing - Parses raw SQL strings into structured abstract syntax trees that capture the semantics of the statement.
- SQL Dialect Adapters - Translates SQL queries between different database dialects by mapping functions, types, and delimiters.
- SQL Lineage Analyzers - Compiles complex SQL statements to trace column-level data lineage and dependencies.
- Column-Level Lineage Extraction - Tracks the flow of data from source tables through transformations to identify column origins.
- Recursive Descent Parsers - Uses a recursive descent parsing engine to translate SQL text into structured syntax trees.
- Lexical Tokenizers - Implements specialized lexers to handle unique keywords and delimiters for various database engines.
- Query Optimizers - Provides automated query optimization by rewriting boolean logic and normalizing predicates to improve execution speed.
- Dynamic SQL Generators - Provides a Python API for programmatically building and modifying dynamic SQL queries.
- SQL Syntax Tree Construction - Provides a programmatic way to build or modify SQL queries by manipulating the abstract syntax tree.
- AST-Based Query Construction - Constructs SQL statements programmatically by manipulating a structured syntax tree via a builder pattern.
- Dialect Specification Frameworks - Provides a framework for creating new SQL engine specifications by overriding core parsing and generation components.
- SQL Generators - Converts a syntax tree back into a SQL string with support for custom formatting and dialect rules.
- SQL Parsers - Parses and transpiles SQL code between different database dialects using structured expression trees.
- AST Transformation Tools - Enables structural modification of SQL queries by searching for and transforming specific nodes within the syntax tree.
- Intermediate Representations - Uses a generic internal representation to translate SQL between different database dialects.
- Semantic Analysis - Resolves semantic context and relationships, such as references within common table expressions.
- Query Planning - Converts optimized syntax trees into directed acyclic graphs of execution steps for query planning.
- Schema Modelers - Represents the structure of tables, views, and columns to enrich the analysis of syntax trees.
- Syntax Validation - Parses SQL strings to identify syntax violations and return structured error messages with precise locations.
- SQL Metadata Extraction - Extracts structural information from queries by identifying the specific columns and tables being referenced.
- SQL Formatters - Generates consistently styled SQL from a syntax tree with options for pretty printing and indentation.
- SQL Query Optimizations - Simplifies the syntax tree by rewriting boolean predicates and normalizing expressions through a set of predefined transformation rules.
- Visitor Pattern Traversers - Implements the visitor pattern to identify and modify specific nodes within SQL expression trees.
- Language Support Extensions - Enables the addition of new database system support through subclassing and plugin registration.
- Expression Transformers - Provides utilities for modifying query logic by traversing and mapping specific node types within the syntax tree.
- Query Result Type Inference - Assigns data types to expression nodes by referencing schema information and function signature metadata.
- Schema-Driven Type Inference - Infers data types for expression nodes using external table definitions and function metadata.
- Database Tools - No-dependency SQL parser, transpiler, and optimizer.
- SQL Development Tools - Transpiler for translating between multiple SQL dialects.
- SQL Parsers - Pure Python SQL parser, transpiler, and builder.
