dbt-labs/dbt-core

Features

• Data Pipeline Orchestration - Provides a system for defining, scheduling, and monitoring complex sequences of data processing tasks and their dependencies within a data warehouse.
• Transformation Frameworks - Provides a command-line framework for transforming, testing, and documenting data within a warehouse using modular SQL and version control.
• Business Metrics - Computes complex business logic by performing mathematical expressions on existing metrics, including time-based offsets and windowing functions to track trends like growth.
• Model Dependency References - Links data models together using references to ensure transformations execute in the correct order.
• Data Transformation - Converts select statements into tables and views within a data warehouse to automate complex data transformation workflows.
• Data Transformation Pipelines - Builds and maintains modular SQL-based data pipelines that transform raw warehouse data into reliable analytical models.
• Transformation Execution - Executes compiled SQL models against databases to transform raw data into structured tables and views.
• Cloud Data Warehouse Connectivity - Establishes secure connectivity between the transformation engine and various cloud-hosted data warehouse platforms.
• SQL Query Execution - Runs standard SQL, Jinja-templated logic, and semantic layer queries against data warehouses with support for CTEs, joins, and aggregations.
• Metric and Dimension Configuration - Enables declarative configuration of business metrics and dimensions to ensure consistent data reporting across diverse analytical tools.
• Semantic Metrics - Standardizes business logic by defining metrics and semantic models in configuration files for consistent calculations.
• Semantic Data Models - Provides a system for defining business logic, metrics, and semantic models that map raw data to consistent organizational definitions.
• Command Line Interfaces - Provides a comprehensive command-line interface for executing data transformation tasks and managing project operations.
• Business Metric Aggregators - Centralizes the definition of business metrics to ensure consistency and reduce code duplication across downstream data tools.
• Local Development Environments - Provides an integrated development environment for writing, testing, and executing data transformations directly within a code editor.
• Incremental Syncing - Updates existing tables by processing only new or changed records using merge or append strategies to reduce compute time.
• Incremental Update Configurations - Provides configuration settings for incremental data updates to optimize warehouse compute costs.
• Impact Analyzers - Visualizes dependency graphs and downstream impacts of model changes to ensure data integrity.
• SQL Templating Engines - Injects variables and macro results into SQL statements using templating to generate flexible and reusable transformation logic.
• Database Dialect Adapters - Translates generic transformation commands into dialect-specific SQL for various data warehouses using an adapter-based abstraction.
• Distributed Data Warehouses - Orchestrates complex data transformation pipelines to optimize compute costs and maintain data freshness across the organization.
• Materialization Strategy Selectors - Determines how data models are physically stored in the warehouse to balance performance and build efficiency.
• Model Performance Analyzers - Extracts start, end, and completion timestamps for model runs to analyze performance and identify bottlenecks in the transformation process.
• Incremental Build Engines - Processes only new or updated records during execution to reduce compute costs and run times.
• CI Workflow Automations - Automates data transformation and testing pipelines upon code changes.
• Data Dependency Managers - Maps downstream reports and dashboards to upstream data models to track lineage and manage complex data dependencies.
• Directed Acyclic Graph Execution Engines - Manages complex data dependencies by building a directed acyclic graph to ensure tasks execute in the correct order.
• Data Modeling Frameworks - Enables declarative definition of data structures and business logic in configuration files that compile into executable code.
• SQL Data Assertions - Creates SQL queries that return failing records to validate data integrity across models.
• Reusable SQL Data Tests - Provides reusable SQL-based validation logic to verify data integrity across multiple resources.
• Automated Lineage Capturers - Automatically extracts and maps column-level dependencies to visualize end-to-end data lineage across the warehouse.
• Lineage Visualizers - Maps and renders the flow of data from raw sources to downstream models and business intelligence dashboards.
• Data Pipeline Orchestrators - Triggers automated workflows based on upstream task completion to manage complex data pipeline dependencies.
• Semantic Query Execution - Runs analytical queries against defined metrics and dimensions to validate calculations and retrieve sample results directly from the warehouse.
• Transformation Impact Previews - Provides automated impact analysis by comparing development code against production state to preview transformation results before deployment.
• Source Data Assets - Registers raw tables loaded by external tools to establish data lineage and enable referencing within transformation models.
• Model Materialization Declarations - Specifies how data models are persisted in the warehouse by declaring output formats within model configurations.
• Data Governance - Provides comprehensive capabilities for managing data lineage, enforcing code quality, and ensuring data transparency.
• Data Integrity and Validation - Executes automated assertions against database tables to verify that data meets defined quality standards such as uniqueness and non-null constraints.
• Data Standardization - Provides utilities to standardize and clean raw source data into consistent, reusable building blocks for downstream models.
• Data Integrity Constraints - Validates data against specified rules during table creation or insertion to ensure data quality.
• Data Dependency Visualizers - Maps dependencies between data models and external assets like dashboards to provide visibility into data flow from source to consumption.
• Data Quality Contracts - Defines explicit schemas and expectations for data models to prevent breaking changes from propagating to downstream consumers.
• Data Quality Frameworks - Executes automated checks against database tables to verify data quality and business logic requirements.
• Database Connection Configurations - Manages warehouse credentials and execution parameters for secure database connectivity.
• Database Connections - Maps distinct warehouse credentials and connection parameters to specific environments to ensure data operations remain separated.
• Dynamic SQL Generators - Embeds conditional statements, loops, and variable references within SQL queries to generate complex, reusable data transformation logic.
• Materialized Views - Creates modular SQL layers that reference source tables to clean and enrich data without materializing intermediate results.
• Transformation Materializers - Configures how transformation results are stored in the warehouse by choosing between tables, views, or incremental updates.
• Metric Query Interfaces - Provides SQL-based interfaces for querying business metrics from the semantic layer.
• Semantic Metadata Retrievers - Offers interfaces for programmatically retrieving semantic model definitions and business metrics.
• Semantic Model Validation - Automates integrity verification for metrics and semantic models within continuous integration pipelines.
• Incremental Table Definitions - Configures target states using SQL statements that the warehouse maintains through incremental refreshes.
• Continuous Integration Validators - Triggers automated data transformation and testing jobs upon code changes to validate models in isolated temporary schemas before merging.
• Data Resource Documentation - Generates searchable catalog sites that list project resources and dependencies to help users understand data provenance.
• Pre-Deployment Validators - Validates data integrity by running automated tests against proposed changes in isolated environments before production merging.
• Cross-Project References - Allows referencing data models from other projects using explicit project-scoped identifiers.
• Project-Wide Materialization Settings - Applies transformation strategies to entire directories of models using central configuration files.
• CI CD Pipelines - Automates the testing, validation, and deployment of data transformation code to ensure high-quality data delivery.
• CI Pipeline Optimizers - Builds and tests only modified models by comparing current state against production to optimize CI execution.
• Data Transformation Validators - Runs continuous integration jobs to validate code changes against production state, ensuring only verified data transformations are merged.
• API Request Authentication - Validates identity using personal or service-level tokens to authorize programmatic access to data transformation resources.
• Transformation Logic Testing - Verifies SQL model transformations against static input datasets to ensure accuracy and reliability of data logic.
• Model Generation Assistants - Generates, refactors, and validates data models using natural language prompts and project-aware context.
• SQL Lineage Analyzers - Compiles SQL statements and traces column-level lineage to visualize dependencies and logic flow.
• Data Pipelines - Transforms data directly within data warehouses.
• Data Pipelines and Orchestration - SQL-based transformation tool for data engineering workflows.
• Data Transformation - Transforms data by managing SQL select statements in warehouses.
• Data Engineering - Framework for transforming data in warehouses using SQL.
• Project Documentation Examples - Uses a friendly description and screenshots of generated docs.
• Analytics Integrations - Links the transformation layer with external analytics tools to provide universal access to governed business metrics.
• Change Data Capture - Captures inserts, updates, and deletes from upstream tables using native change data capture mechanisms to keep downstream models synchronized.
• Change Detection Engines - Identifies modifications in source data using timestamp-based tracking or column-level value comparisons to trigger historical record creation.
• Freshness Monitoring - Monitors and reports the recency of raw data sources to ensure downstream transformations use up-to-date information.
• Data Access & Abstraction - Provides an abstraction layer to define relationships between source system tables and staging models using macros.
• Data Exploration - Provides a unified interface to query data, view project metadata, and navigate lineage for exploratory analysis.
• Catalog Integration - Registers table metadata in external Iceberg catalogs during materialization to ensure data objects are discoverable across different compute engines.
• Model Column Specifications - Explicitly declares data types for columns within models to enforce schema consistency.
• Uniqueness Enforcement - Identifies specific rows to enable upsert behavior, ensuring existing records are updated instead of duplicated.
• Data Models - Treats data models as stable interfaces to allow for graceful evolution, testing, and deprecation of data products.
• Data Schema Management - Handles automated column additions or removals in source data to prevent pipeline failures.
• Schema Versioning - Maintains multiple concurrent versions of a data model to allow for safe transitions between schema or logic updates.
• Historical - Implements slowly changing dimensions to record changes to source data over time and track historical state.
• Data Quality Metadata Discoverers - Accesses information about model lineage, test results, and source freshness to help users understand data structure and quality.
• Load-Time Type Casting - Specifies explicit data types for columns during the loading process to ensure data consistency.
• Record Updates - Updates existing rows in the warehouse based on a unique identifier instead of appending duplicates when new data arrives.
• Event Timestamp Definitions - Specifies occurrence times for data records to enable incremental processing and advanced dataset comparison.
• Graph Querying - Maps business metrics and dimensions into a unified graph structure to enable consistent data querying.
• Semantic Manifest Generators - Parses project files to create a structured representation of data relationships and metrics that serves as the foundation for semantic query execution.
• SQL-Based Semantic Layer - Enables data exploration through a semantic layer that abstracts complex SQL logic.
• Parallel Data Transformation - Executes multiple nodes of a data transformation graph concurrently to reduce total project runtime by adjusting active threads.
• Base Model Composition - Combines raw source tables through joins or unions to prepare unified concepts before applying staging transformations.
• Reusable Logic Definition - Encapsulates reusable SQL snippets or database functions to standardize complex calculations across multiple models.
• SQL Dialect Adapters - Writes transformation logic using specific SQL syntax supported by the underlying database engine to leverage platform-specific features.
• Full Refresh Strategies - Overrides incremental logic to rebuild a model from scratch, ensuring the table state matches the source data.
• Incremental Build Engines - Targets only modified data models and their downstream dependencies during execution to minimize computational overhead.
• Environment Comparators - Analyzes differences between production and development environments to verify the impact of changes before deployment.
• Manifest-Driven Dependency Managers - Maintains a JSON-based project state file to track model definitions, lineage, and execution history across environments.
• Project Organization Tools - Structures large-scale data transformations into independent, reusable components shared across teams.
• Project Configuration Managers - Centralizes project metadata and environment connections to manage transformation task execution.
• State-Aware Job Orchestrators - Detects code or data changes automatically to determine which models require rebuilding during job execution.
• Single Sign-On - Enforces centralized authentication through single sign-on providers.
• Coding Standards Enforcement - Integrates automated linting to enforce SQL coding standards during the build process.
• Incremental Scan Optimizers - Optimizes performance for large datasets by processing only new or changed records using incremental batching.
• Metric Validation Tools - Executes metric definitions against a data warehouse during development to verify calculations and inspect results before deployment.
• Test Execution Overrides - Allows setting default execution parameters for tests while supporting individual instance overrides.
• Structural Versioning Enforcers - Validates that model output schemas remain consistent and prevents breaking changes through explicit versioning.
• Trace Metric APIs - Enables programmatic access to defined metrics for custom data workflows and reporting.
• Data Model Groups - Allows grouping related data nodes into logical units to simplify management within complex architectures.
• Atomic Model Replacements - Ensures consistent data access by replacing existing models with new versions atomically.
• Freshness Analyzers - Queries metadata to verify that data sources and models meet freshness expectations.
• Column-Level Lineage Extraction - Maps end-to-end flow of data for individual columns across models and sources to provide provenance visibility.
• Query Result Caching - Stores precomputed query results in materialized views that refresh automatically to accelerate read performance.
• Incremental Data Merging - Merges historical data from incremental tables with recent events to provide a unified dataset for reporting.
• Data Quality Monitors - Tracks test results and freshness to verify data accuracy and trigger alerts.
• Custom Schema Configurations - Organizes transformed data into specific database schemas outside of the default target location to manage warehouse structure.
• Periodic Snapshots - Executes periodic snapshots to capture and preserve historical states of changing data.
• Transformation Schedulers - Executes data transformation pipelines on short, recurring intervals to ensure downstream reporting reflects the most recent information.
• Data Type Casting - Forces specific data types on columns during transformation to ensure consistency across downstream models.
• Warehouse Grant Configurations - Allows provisioning of specific access permissions for users and groups on data warehouse tables via configuration.
• Macro-Based Alias Resolvers - Defines programmatic logic for naming database objects by overriding default resolution macros.
• Database Lifecycle Hooks - Runs custom SQL commands before or after model execution by embedding templated strings that resolve to database operations.
• Table Metadata Inspection - Provides a searchable interface to inspect tables, columns, metrics, and dimensions within the data warehouse.
• Warehouse Schema Provisioning - Automates the creation of target schemas within the warehouse to ensure isolated development spaces.
• Implicit Relationships - Links datasets implicitly through shared entity keys to allow joining information without requiring explicit join definitions.
• Version-Controlled Configuration Stores - Stores and manages standalone SQL statements within a project directory to enable version control and collaboration.
• Metric Calculators - Calculates the relationship between two distinct measures by dividing a numerator metric by a denominator metric to produce a ratio.
• Cumulative - Aggregates values over a specified time window to track running totals of a measure across the entire period.
• Execution Scoping - Isolates validation tasks to individual tables or sources to reduce execution time and focus troubleshooting efforts.
• Macro-Based Database Naming - Overrides default naming logic for model databases using custom macros.
• Resource Naming Overrides - Overrides default database, schema, or object names to match specific platform naming conventions.
• Schema Naming Macros - Overrides default naming conventions for database schemas using custom macros.
• Metric Query Languages - Supports querying multi-dimensional metrics via graph-based interfaces.
• Soft Deletion Tracking - Captures the removal of source data by inserting new records into snapshot tables to maintain historical state.
• Development Schema Isolation - Assigns unique target schemas to individual users to prevent concurrent development conflicts.
• Seed Schema Mappings - Directs the placement of imported data files into specific warehouse schemas to maintain organized data structures.
• Table Data Processing - Persists transformed results as physical rows on disk to improve query performance and reduce compute costs for frequently accessed datasets.
• Build Performance and Optimization - Reduces warehouse costs and execution time by sampling data, reusing previously built models, and testing only changed components.
• Downstream Dependency Mappers - Maps data models and sources to downstream analytical exposures to provide visibility into how data is consumed.
• Resource Annotators - Attaches descriptive metadata and documentation to data models and sources to provide business context.
• Incremental Execution - Compares current changes against production artifacts to identify and process only modified models, reducing execution time and costs.
• Merge Deployment Automators - Executes data transformation workflows automatically when code changes are merged to ensure production environments remain synchronized.
• Development Workflow Automation - Uses artificial intelligence agents to generate, refactor, and validate data transformation code directly within the development environment.
• Lifecycle Automation Hooks - Triggers custom SQL operations at specific stages of the transformation lifecycle.
• Linting Workflows - Enforces coding standards and identifies syntax errors in SQL files during continuous integration.
• Parallel Execution - Configures concurrent thread counts to process dependency graphs and execute independent tasks simultaneously.
• Project Metadata Analyzers - Generates machine-readable project state and lineage files after job executions.
• Metadata Browsers - Provides an interface to explore project resources, metrics, and dimensions with integrated lineage and context.
• Build Optimization - Detects code and data changes to skip unnecessary model rebuilds and trigger rebuilds when warehouse tables are missing.
• Dependency Visualizers - Renders graphical representations of data dependencies and transformation flows to help navigate complex project structures.
• Execution Deferrers - Enables deferring execution to specific deployment states to ensure consistency across environments.
• Resource Selection Rules - Enables targeting individual models or dependencies for execution using selection syntax to isolate parts of the data graph.
• Macro Logic Overrides - Allows redirecting calls to specific macros to custom implementations for project-specific behavior adjustments.
• Identity and Access Management - Integrates with enterprise identity providers to secure platform access via SSO and service principals.
• Identity Synchronization - Automates the synchronization of user identities and group memberships from external providers.
• Automated Code Quality Tools - Detects syntax and compilation errors in SQL files during automated builds to ensure code quality.
• Build Decision Explainers - Provides detailed explanations for why specific models were rebuilt or reused to help troubleshoot state-based execution logic.
• Execution Metadata - Tracks status and timing of transformation runs to distinguish between configuration and materialized warehouse state.
• Definition Validators - Verifies the integrity of metric definitions and dimensions by parsing project manifests before execution.
• Test Failure Debugging Tools - Traces errors to their origin by identifying upstream columns contributing to failing tests or unexpected data behavior.
• Validation Threshold Configurations - Defines acceptable error limits for data tests to determine when validation failures trigger alerts versus remaining within tolerable bounds.
• SQL Query Validators - Analyzes SQL code during development to identify syntax errors and dialect-specific issues before execution.
• SQL Logic Validators - Translates project models, tests, and macros into executable SQL code without executing them against the database.
• Integration APIs - Exposes defined business metrics programmatically via standard interfaces for integration into custom applications.
• Third-Party API Integrations - Connects the semantic layer to third-party reporting platforms for governed data exploration and visualization.
• Cross-Project - Aggregates metadata across multiple projects to provide a unified view of data dependencies and flow.
• Model State Auditors - Retrieves execution history and configuration status to track deployment health and recent changes.
• Natural Language Query Interfaces - Interacts with semantic models and data insights using natural language to retrieve information powered by the semantic layer.
• Circular Dependency Resolutions - Detects and prevents circular references between data models to ensure stable execution across distributed environments.
• Cloud Authentication - Connects the local environment to a remote platform using credential files to enable integrated development and management.
• Metadata Query Interfaces - Provides interfaces for querying warehouse metadata and lineage remotely.
• Conversion Rate Metrics - Measures the frequency of secondary events occurring after primary events to track user behavior.
• GraphQL API Generators - Exposes project and model metadata through automatically generated GraphQL interfaces.
• Ingestion Lookback Managers - Applies a lookback window to incremental filters to capture records that arrive after their expected ingestion time, ensuring data consistency.
• Software-Defined Architectures - Extends the framework with user-defined logic for persisting data models in the warehouse.
• Microbatch Processors - Segments large datasets into temporal windows to optimize performance and enable efficient incremental updates.
• Analytical Data Converters - Transforms retrieved tabular data into analytical structures for integration with data processing libraries.
• Data Modeling Standards - Enforces consistent naming and structural patterns across data models to improve maintainability.
• Intermediate Representations - Separates complex, multi-step data processing into dedicated models to simplify final reporting tables.
• Data Query Management - Stores and retrieves frequently used query definitions to ensure consistent reporting across teams.
• Downstream Exposure Refreshers - Maps data models to downstream dashboards and triggers automatic refreshes during scheduled transformation jobs.
• Snapshot Hooks - Triggers custom SQL commands or operations before or after the snapshot process to manage database state.
• External Data Integrations - Supports community-maintained plugins for executing transformation logic across diverse third-party data warehouses and query engines.
• Metadata Inspection Tools - Provides programmatic access to metrics, dimensions, and saved queries for discovery and documentation purposes.
• Semantic Definition Organization - Groups metrics and semantic models alongside transformation logic to maintain consistent structure and improve developer navigation.
• Query Caching Strategies - Accelerates query performance by storing frequent semantic layer query results in local memory.
• Previewers - Allows developers to execute and preview metric queries directly against the warehouse during the development process.
• Job Performance Analyzers - Analyzes job performance by breaking down execution metrics to identify bottlenecks and optimize resource usage.
• Model Metadata Retrieval - Retrieves historical run information and materialization status for specific data models to track project performance.
• Schema Reconciliation - Automatically updates snapshot tables to ensure historical data remains consistent with current source structures.
• Schema Assignments - Assigns data models to specific database schemas to maintain organized and accessible data structures.
• Analytical SQL Compilers - Processes SQL files containing project-specific references into executable code for use in external data visualization or reporting tools.
• Iceberg Table Materializers - Transforms and stores data using the open-source table format to ensure portability and interoperability across different data engines and infrastructure.
• Build Performance Optimizers - Identifies unchanged logic and data to skip redundant builds or clone existing objects, reducing compute time.
• Build Performance Analyzers - Records execution duration and status for models to identify performance bottlenecks during transformation runs.
• Resource Performance Analyzers - Examines historical execution metrics to optimize orchestration configurations and reduce infrastructure costs.
• Version Migration Utilities - Enables incremental testing and adoption of new model versions while pinning dependencies to stable releases.
• Metadata Querying - Retrieves structured metadata about transformation assets to support discovery and auditing within the project.
• Project Documentation - Compiles project metadata and lineage into a searchable documentation site.
• Project Resource Metadata Explorers - Retrieves metadata, lineage, health status, and performance history for models, sources, and tests.
• Version Control Integrations - Integrates version control systems to enable tracking and collaborative development of transformation logic.
• Build Automation - Automates the full lifecycle of data tasks including building, testing, and shipping by leveraging project lineage and defined data models.
• Incremental Compilation Strategies - Enables project-specific incremental data update logic through custom macros.
• Configuration Validation - Verifies project configuration files against defined schemas to ensure syntax correctness before execution.
• Environment Configuration Management - Enables defining multiple deployment environments within a single configuration for context-aware execution.
• API-Driven Task Triggers - Allows programmatic initiation of data transformation tasks via API for integration with external CI/CD pipelines.
• Job Event Notifications - Sends automated notifications when data transformation jobs or tests fail to ensure rapid response to pipeline issues.
• Job Output Retrievers - Enables triggering automated workflows, listing execution history, and retrieving build artifacts through administrative interfaces.
• Job Workflow Administrators - Triggers, monitors, and manages job runs, including retrying failures and retrieving execution artifacts.
• Version Control Repository Integrators - Connects development environments to version control repositories to manage files and automate CI workflows.
• Full Refresh Orchestrators - Drops and recreates existing tables from scratch to ensure historical data reflects the most recent transformation logic.
• Access Control - Codifies database permissions directly within resource configurations for version-controlled security.
• Model Access Controls - Limits data model visibility to specific groups to prevent unauthorized downstream references.
• Permission-Based Security - Automates the application of security policies by configuring database grants as structured definitions.
• Model Access Governance - Restricts project and user access to specific data models to maintain data security.
• Personal Access Tokens - Enables generation of account-scoped personal tokens for programmatic access to transformation services.
• Data Asset Metadata Exposers - Provides access to model, metric, and column metadata to power data catalogs and downstream analytics applications.
• Live Transformation Verification - Compiles and executes code against the warehouse during development to detect errors and verify logic before committing changes.
• Project Network Connectors - Links separate data transformation projects into a unified network to manage complexity and enforce governance across domains.
• Task Retry Policies - Supports automatic retry strategies for failed tasks to improve pipeline reliability and save compute resources.
• System Activity Auditors - Maintains historical records of system events to track actions and ensure organizational accountability.
• Execution History Auditors - Tracks and audits the historical performance and outcomes of data model executions.
• Execution Logs - Provides access to SQL execution logs and process history for troubleshooting pipelines.
• Pipeline Health Monitors - Evaluates the operational health of models and sources based on build and test status.
• Batch Retry Mechanisms - Re-runs only failed time-bounded batches to save resources and time.
• Execution Metadata - Queries historical and current execution details for transformation tasks to track progress and status.
• Target Execution Runners - Enables selection of specific model subsets for execution or testing to isolate changes and optimize resources.
• Performance Monitoring - Analyzes execution timing and performance trends across data models.
• Downstream Usage Mappers - Maps data models to external consumers like dashboards to track dependencies and provide context for stakeholders.
• Formatting and Linting Pipelines - Standardizes SQL, YAML, and other project files using integrated formatting and linting pipelines.
• Warehouse State Exposers - Queries the actual metadata and status of objects currently existing in the data warehouse following a transformation execution.
• Project Consistency Analyzers - Analyzes project structure and configuration against established standards to identify deviations.
• Test Execution Optimizers - Reduces warehouse costs by aggregating multiple tests into single queries and reusing previous test results when logic remains unchanged.

Star-Verlauf