yahoo/kafka-manager
Kafka Manager
Kafka Manager is a web-based management interface and monitoring tool for Apache Kafka clusters. It serves as a central control plane for topic administration, consumer monitoring, and cluster health inspection.
The project provides specialized utilities for data rebalancing and partition reassignment to distribute workloads across brokers. It also includes tools to optimize partition leadership by electing preferred replicas.
The platform covers a broad range of administrative capabilities, including the creation and configuration of message topics, tracking of consumer offsets, and the collection of broker performance metrics. Access to these operations is managed through identity validation and role-based restrictions via directory servers.
Features
- Kafka Cluster Administration - Provides a centralized interface for managing Kafka brokers and topics to maintain cluster health.
- Consumer Lag Monitoring - Tracks consumer offsets and group progress to identify lag and ensure reliable message processing.
- Cluster Coordination - Interacts with Zookeeper to manage topic configurations and track broker registration and health across the cluster.
- Partition Reassignments - Generates and executes new partition assignments across brokers to balance workload and optimize performance.
- Cluster Health Monitoring - Monitors topics, consumers, offsets, and broker distribution across multiple clusters to ensure stability.
- Shard Rebalancing - Moves data partitions between brokers to ensure even workload distribution and optimize performance.
- Topic Configuration - Provides tools to create, delete, and update Kafka topic configurations and partition counts.
- Kafka Management Interfaces - Provides a web interface for monitoring and managing Apache Kafka clusters, including topics and consumers.
- LDAP Authentication - Validates user credentials against directory servers using LDAP for centralized authentication and access restrictions.
- JMX Metric Collectors - Polls Java Management Extensions (JMX) beans to retrieve real-time performance data from remote Kafka brokers.
- Distributed Leader Election - Runs elections to determine which replica should lead a partition to optimize cluster performance.
- Kafka Permissions Management - Manages user identities and restricts access to Kafka clusters based on directory server groups.
- Role-Based Access Control - Restricts management operations by validating user group memberships against an external directory server via role-based access control.
- Partition Reassignment Tools - Provides a dedicated utility to generate and execute data partition assignments to balance broker workload.
- Message Brokers - Retrieves and displays broker and topic level performance metrics using management beans.
- Data Ingestion Pipelines - Management interface for Apache Kafka clusters.
