prometheus-operator/kube-prometheus
Kube Prometheus
kube-prometheus is a monitoring stack deployment and orchestration framework. It uses an operator pattern to automate the installation and lifecycle management of Prometheus and Alertmanager via custom resource definitions.
The project focuses on scaling data collection through hash-based target sharding and topology-aware distribution to reduce cross-zone traffic. It implements a sidecar-based configuration reloading mechanism and utilizes consistent hashing to distribute scrape targets across multiple instances.
The system covers broad observability capabilities including metric data collection, distributed alerting rule evaluation, and alert notification routing. It manages data persistence through dynamic storage provisioning and ensures long-term data availability via object storage backups and remote write configurations.
The implementation is primarily written in Jsonnet.
Features
- Operator-Based Lifecycles - Automates the deployment, scaling, and upgrading of monitoring stacks using the Kubernetes Operator pattern.
- Observability Stack Deployments - Automates the installation and management of a full monitoring stack, including Prometheus, Alertmanager, and associated dashboards.
- Push-Based Controller Topologies - Distributes scrape targets based on network topology or zones to reduce cross-zone traffic.
- Topology Aware Sharding - Pins shards to specific zones to restrict scraping to local targets and reduce network traffic.
- Automated Target Configuration - Generates target configurations automatically using label selectors instead of manual configuration files.
- Dynamic Scrape Target Discovery - Identifies monitoring targets for scraping using label selectors and intermediary services.
- Scrape Target Sharding - Splits scrape targets into groups assigned to different instances to scale collection capacity.
- Target Distribution - Splits scrape targets across multiple instances using hash-based distribution to scale data collection.
- Automatic Monitoring Shard Scaling - Adjusts the number of monitoring instances automatically to distribute targets across shards based on workload.
- Custom Resource Mappings - Translates high-level Kubernetes custom resources into Prometheus and Alertmanager configuration files using the operator pattern.
- Monitoring Stack Automation - Automates the installation and lifecycle of Prometheus and Alertmanager using custom resource definitions.
- Monitoring Shard Scaling - Implements hash-based target sharding to distribute scrape workloads across multiple Prometheus instances for scalability.
- Alertmanager Deployment - Orchestrates the deployment of high-availability Alertmanager clusters integrated with custom configuration objects.
- Lightweight Metrics Agents - Runs lightweight instances focused on scraping and forwarding metrics without utilizing local storage.
- Alerting Integrations - Connects monitoring instances to an alerting cluster to automatically discover and forward triggered alerts.
- Service Discovery Integrations - Integrates with cloud providers and DNS to identify monitoring targets across diverse environments.
- Hash-Based Data Distribution - Distributes scrape targets across multiple Prometheus instances using consistent hashing to scale data collection.
- Alert Notification Management - Deduplicates alerts and routes grouped notifications to external integrations such as email or chat.
- Alert Routing - Configures how alerts are grouped, inhibited, and routed to specific receivers based on matching metadata rules.
- Label-Based Targeting - Automatically discovers monitoring targets by matching Kubernetes resource labels against predefined selector patterns.
- Pull-Based Metric Scraping - Implements a pull-based mechanism to collect telemetry from Kubernetes pods using service discovery.
- Monitoring Rule Definitions - Creates rules for triggering alerts or pre-calculating metrics based on discovered system data.
- Scrape Configuration Automators - Automates the generation of scrape configurations for various targets using custom resource definitions.
- Monitoring Instance Provisioning - Sets up required permissions and service accounts to discover and scrape targets within a cluster.
- Pod Discovery - Finds and configures monitoring for specific pods using Kubernetes label selectors.
- Prometheus Cluster Management - Automates the deployment and scaling of Prometheus instances using cluster-wide stateful sets or node-local scopes.
- Service Discovery - Identifies active Kubernetes pods and services as monitoring endpoints using label selectors.
- Data Forwarders - Streams collected metrics to remote long-term storage without requiring local evaluation.
- Data Persistence - Ensures metric data survives pod restarts and rescheduling by coupling pods with persistent volume claims.
- Shard Data Preservation - Retains pods from scaled-down shards so historical metric data remains queryable until the retention period expires.
- Custom Scrape Targets - Configures external monitoring targets or complex scrape patterns that exceed standard service monitor capabilities.
- Sharding Override Controls - Determines which shard handles a target by overriding the default sharding hash with specific labels.
- Dynamic Rule Management - Supports the dynamic loading of alert and recording rules without requiring a process restart.
- High-Availability Instance Redundancy - Runs multiple identical monitoring instances with unique labels to ensure high availability and eliminate single points of failure.
- Remote Write Configuration Delegation - Allows users to define their own remote write settings via scoped resources without requiring administrative access.
- Storage Provisioning - Integrates with storage classes to automatically create and mount high-performance volumes across cloud providers.
- Health Status APIs - Exposes operational status, replica counts, and health conditions via a dedicated API subresource.
- Zonal Target Deployment - Generates node selectors to deploy instances within the same zone as the targets they monitor.
- Alert Notification Routing - Prometheus Operator configures routing, inhibition rules, and custom receivers for alert notifications.
- Object Storage Backups - Streams time-series data to external object storage targets to enable global querying and long-term retention.
- Configuration Hot-Reloading - Uses a sidecar container to monitor configuration changes and trigger API reloads without restarting the main process.
- External Service Monitoring - Probes ingresses and static endpoints using a prober service to monitor external availability.
- Metric Data Preservation - Preserves metric samples during network outages or pod evictions by utilizing persistent storage.
- Alerting Rule Validators - Implements validation checks to ensure monitoring rules are syntactically and semantically correct before application.
- PromQL-Based Alerting and Recording Rule Evaluation - Executes PromQL alerting and recording rules against a distributed query API to generate alerts.
- Distributed Rule Processing - Executes recording and alerting rules across multiple instances using a centralized ruler for scalability.
- Rule Evaluators - Deploys specialized instances to evaluate monitoring rules via compatible API endpoints.
- Reusable Security Profiles - Defines reusable TLS and authorization profiles to avoid redundant configuration across monitoring resources.
- Node-Local Monitoring Agents - Runs monitoring agents on every node to optimize metric collection from local targets and reduce network traffic.
- Scrape Job Aggregation - Aggregates scrape configurations using service discovery mechanisms like DNS, cloud providers, and cluster APIs.
- Agent-Mode Metric Forwarding - Provides a lightweight agent deployment that collects metrics and forwards them to remote storage without utilizing local long-term storage.
- Remote Write Resource Constraints - Imposes constraints on queue capacity and metadata transmission to maintain cluster stability during remote writes.
- Node-Local Target Filtering - Ensures agents only target pods on the local node using field selectors to reduce API load.
