ELT Data Pipeline Tools

Airbyte is a data integration platform designed to synchronize information between diverse applications, databases, and data warehouses. It functions as an extract, transform, and load orchestrator that manages automated data movement workflows across cloud, on-premise, and hybrid environments. The platform provides a standardized interface for connectors, enabling the movement of structured and unstructured data while maintaining stateful checkpoints for reliable incremental syncing. The platform distinguishes itself through a containerized architecture that isolates connectors to prevent de

Meltano is an open-source platform for building, running, and orchestrating ELT (Extract, Load, Transform) data pipelines. It provides a declarative, YAML-driven configuration system that defines entire pipeline workflows, including data connectors, schedules, and transformations, without requiring imperative code. The platform is built on the Singer specification for data connectors and integrates with dbt for SQL-based transformations and Apache Airflow for scheduling and orchestration. What distinguishes Meltano is its comprehensive approach to pipeline management, combining a curated cata

DataHerald is a natural language text-to-SQL interface and data analytics tool that translates English questions into executable database queries using a large language model. It functions as a query generator and connectivity layer capable of retrieving database records and summarizing the results into human-readable explanations. The project features a dedicated business logic layer for defining database constraints and managing verified query pairs to improve translation accuracy. It acts as a multi-warehouse data connector, allowing for the indexing and querying of multiple SQL databases

Great Expectations is a data quality testing framework and observability platform designed to monitor the reliability of data pipelines. It provides a structured environment for defining, documenting, and automating data quality assertions, allowing teams to validate datasets against expected structure and content before they move through downstream processes. The project distinguishes itself through a declarative domain-specific language that stores quality rules as version-controlled configuration files. It utilizes an execution engine abstraction to translate these high-level assertions in

Records is a SQL database client designed for executing raw queries and managing result sets through a simplified interface. It provides a parameterized SQL executor to bind values to placeholders, ensuring safe data handling and preventing injection attacks, alongside a database transaction manager for grouping operations into atomic units. The project includes a dedicated command-line interface for running database statements and exporting query results directly to local files. This tooling allows for the conversion of SQL result sets into multiple serialization formats, including CSV, JSON

DevLake is a DevOps data platform and analytics tool designed to orchestrate data pipelines that ingest, transform, and sync metadata from external development tools into a unified database. It functions as a system for collecting and normalizing data from source control, CI/CD pipelines, and issue trackers into a standardized schema to enable consistent software delivery analytics. The platform distinguishes itself by transforming tool-specific data into a common domain model, allowing for the calculation of engineering metrics via SQL. It provides specialized frameworks for measuring DORA m

dlt is a Python data ingestion tool and ETL pipeline framework designed to fetch data from diverse sources and persist it into structured destinations. It functions as a schema inference engine that automatically detects data types and flattens nested JSON structures into relational tables, moving data from sources to lakehouses, warehouses, or vector databases. The project distinguishes itself through AI-powered pipeline generation, using large language models to scaffold extraction code and connectors for REST APIs. It also supports multimodal vector storage and specialized population of ve

This project is a comprehensive framework for engineering financial data pipelines, designed to automate the collection, cleaning, and synchronization of large-scale market datasets. It functions as a quantitative trading data engine, providing the infrastructure necessary to manage historical and real-time asset pricing information for research and machine learning workflows. The system distinguishes itself through a configuration-driven approach to orchestration, allowing users to manage complex data acquisition tasks across multiple financial providers. It features resilient middleware tha

Dapper is a lightweight object-relational mapper for .NET that functions as a high-performance data access library. It operates by extending standard database connection interfaces, allowing developers to execute raw SQL queries while automating the mapping of database results to strongly-typed objects. The library distinguishes itself through its use of runtime code generation, which creates high-performance instructions to map database rows to object properties with minimal overhead. It provides flexible data retrieval options, supporting both memory-buffered loading for speed and row-by-ro

graphql-engine is an automated GraphQL API engine that transforms database tables and relationships into a queryable GraphQL schema. It functions as a federation gateway and mapper, instantly generating APIs with built-in filtering, pagination, and mutations from existing databases and remote schemas. The project distinguishes itself through a fine-grained access control layer that enforces row-level and field-level permissions. It further provides a real-time data subscription server that converts standard queries into live streams and a system for triggering event-driven webhooks and notifi

Dagster is a data orchestration platform designed to manage the entire lifecycle of data assets through declarative modeling and version-controlled code. It functions as a workflow engine that treats data assets as first-class primitives, allowing teams to define, schedule, and monitor complex pipelines while maintaining clear visibility into lineage, dependencies, and data quality. The platform distinguishes itself by using a code-as-configuration framework that enables standard software engineering practices, such as unit testing and local mocking, to be applied directly to data workflows.

dbt-core is a command-line framework for transforming data within a warehouse using modular SQL and version control. It functions as a data transformation engine that enables users to define data structures and business logic through declarative configuration files, which the system then compiles into executable code. By managing complex data dependencies through a directed acyclic graph, it ensures that transformation tasks execute in the correct order while maintaining a manifest-driven state to track lineage and execution history. The project distinguishes itself through an adapter-based d