apache/incubator-devlake
Incubator Devlake
DevLake is a DevOps data platform and analytics tool designed to orchestrate data pipelines that ingest, transform, and sync metadata from external development tools into a unified database. It functions as a system for collecting and normalizing data from source control, CI/CD pipelines, and issue trackers into a standardized schema to enable consistent software delivery analytics.
The platform distinguishes itself by transforming tool-specific data into a common domain model, allowing for the calculation of engineering metrics via SQL. It provides specialized frameworks for measuring DORA metrics, analyzing engineering throughput, and tracking open source community engagement and contributor health.
The system covers a broad range of capabilities including plugin-based data ingestion, incremental synchronization to reduce API load, and the creation of custom engineering dashboards. It supports data pipeline orchestration to automate the movement of information from diverse external sources into a centralized relational database.
Deployment is managed as a cloud-native application using Helm charts for Kubernetes environments.
Features
- Data Ingestion Pipelines - Provides a comprehensive framework for extracting, transforming, and loading DevOps metadata from diverse sources.
- Software Delivery Analytics - Provides a comprehensive platform for measuring DORA metrics and community health by aggregating activity across the software development lifecycle.
- Domain-Model Normalizations - Transforms tool-specific data into a standardized common schema to allow cross-platform metric calculation.
- Data Standardization - Transforms and normalizes disparate data formats from various development tools into consistent structures.
- Incremental Syncing - Implements incremental synchronization to fetch only new or changed records, reducing API load.
- Unified Schema Syncs - Aggregates data from multiple external APIs into a consistent, unified schema for a full lifecycle view.
- Data Normalization Layers - Provides a transformation layer that converts tool-specific data into a common internal schema.
- Data Pipeline Configurations - Implements a configuration system for defining connection scopes and transformation rules to automate external data collection.
- Data Pipeline Orchestration - Sequences data collection and enrichment tasks through a configurable workflow to automate data movement.
- Plugin-Based Ingestion - Uses a modular plugin architecture to ingest metadata from diverse external development tools.
- DevOps Metadata Ingestors - Provides specialized plugins for the automated gathering of metadata from source control and CI/CD tools.
- DevOps Metric Collection - Provides a centralized system for gathering activity metrics from issue trackers, source control, and CI/CD pipelines.
- External Data Connectors - Implements a flexible framework for linking external tools and managing the collection and transformation of incoming data.
- SQL Metric Definitions - Calculates engineering performance indicators by executing SQL queries against a normalized relational database.
- Delivery Performance Tracking - Measures cycle time and deployment frequency to identify bottlenecks in the release process.
- DevOps Data Platforms - Collects and normalizes data from source control, CI/CD, and issue trackers into a standardized schema.
- Schema-Standardizing Transformations - Provides a set of custom rules for converting raw API responses into a standardized layer for accurate metric computation.
- Normalization Engines - Normalizes diverse input formats from multiple software development tools into a unified structure for querying.
- DORA Metrics - Implements a framework for tracking DORA metrics to evaluate software release efficiency.
- Engineering Throughput Analytics - Provides tools for measuring cycle time and throughput across teams to identify bottlenecks and track velocity.
- Development Velocity Metrics - Tracks engineering performance and delivery velocity by analyzing data from version control and CI/CD tools.
- Data Ingestion Plugins - Uses a modular plugin system to fetch raw data from diverse external APIs and toolchains.
- Connection Management - Provides a dedicated interface for managing links and credentials between external tools and the system.
- Data Processing Configurations - Offers a guided process for setting up ingestion parameters to automate how data is gathered from various sources.
- Logical Project Grouping - Organizes data scopes from multiple tools into single logical projects for unified performance analysis.
- Data Querying - Provides an interface for querying normalized DevOps data collections using SQL.
- Data Schema Management - Provides tools for modifying the domain layer schema and transforming data models to fit specific organizational needs.
- Git Repository Analysis - Tracks component and file-level metrics to identify high-churn areas and codebase complexity within Git repositories.
- REST APIs - Provides a standard RESTful API enabling programmatic retrieval and modification of system data for automation.
- Webhook - Receives real-time data pushes from external services via HTTP endpoints when native polling is unavailable.
- Plugin Development Kits - Provides a software development kit for building custom integration plugins while managing external API rate limits.
- Webhook Integrations - Implements a capability for receiving real-time data pushes from external tools via HTTP webhooks.
- Trend Visualizations - Provides visualizations of bug patterns and resolution timelines to analyze quality issues.
- Community Engagement Analytics - Periodic aggregation of activity data to evaluate trends and growth in community contributions over time.
- Open Source Community Analytics - Gathers contributor and engagement metrics to evaluate the health and growth of community-driven projects.
- Engagement Analytics - Tracks engagement and contributor health metrics to understand user interaction with open source projects.
- Metric Dashboards - Ships prebuilt dashboards for tracking delivery process performance indicators and development progress.
- Custom Metric Dashboards - Enables creation of user-defined SQL visualizations to monitor development KPIs and team goals.
