didi/KnowStreaming
KnowStreaming
KnowStreaming is a centralized Kafka cluster management platform that unifies multi-cluster federation, load balancing, disaster recovery, and resource governance through a web-based graphical interface. It provides a single control plane for administering brokers, topics, partitions, consumer groups, ACLs, and connectors across heterogeneous Kafka clusters without requiring CLI commands or agent deployment on brokers.
The platform distinguishes itself through automated load balancing that redistributes partition leaders and replicas to eliminate hotspots and improve throughput, combined with a multi-active disaster recovery orchestrator that coordinates failover and traffic switching between clusters to maintain availability during outages. It also includes a replica migration engine for moving topic replicas between brokers or racks with zero downtime, and supports scaling replicas on the fly to adjust fault tolerance and read capacity.
Beyond these core capabilities, KnowStreaming offers real-time monitoring of cluster health and metrics through a non-intrusive polling pipeline, automatic cluster discovery and registration for any Kafka version, and graphical governance of topic lifecycles, partition assignments, and configuration policies. The platform is built on a plugin-based agentless architecture that extends management capabilities through hot-pluggable modules without modifying broker configurations or restarting clusters.
Features
- Kafka Cluster Administration - Manages brokers, topics, partitions, and consumer groups across multiple Kafka clusters through a graphical interface.
- Kafka Management Interfaces - Ships a web-based graphical interface for administering Apache Kafka clusters with zero-instrumentation deployment.
- Cluster Load Balancing - Provides automated rebalancing of partition leaders and replicas across brokers to eliminate hotspots.
- Kafka Partition Balancers - Redistributes partition leaders and replicas across brokers to eliminate hotspots and improve cluster throughput.
- Partition Leader Redistributors - Automatically redistributes partition leaders and replicas across brokers to eliminate hotspots and improve throughput.
- Multi-Active Failover Coordinators - Coordinates failover and traffic switching between Kafka clusters to maintain availability during outages.
- Partition Reassignment Engines - Executes partition reassignment plans by coordinating leader election and data copying across brokers with zero downtime.
- Topic Replica Movers - Moves topic replicas between brokers or racks to rebalance storage and network load without downtime.
- Kafka Resource Controllers - Controls topic lifecycle, partition assignments, and configuration policies across clusters from a centralized dashboard.
- Kafka Resource Orchestrators - Translates UI-driven operations into Kafka administrative API calls for topic, partition, and ACL management.
- Cross-Cluster Federation - Unifies heterogeneous Kafka clusters under a single control plane by abstracting version-specific protocols and metadata.
- Cross-Cluster Failover Orchestrators - Coordinates failover and traffic switching between Kafka clusters to maintain availability during outages.
- Disaster Recovery Orchestrators - Coordinates failover and traffic switching between Kafka clusters to maintain availability during outages.
- Kafka Replica Scalers - Adds or removes replicas for a topic to adjust fault tolerance and read capacity on the fly.
- Topic Replica Managers - Migrates and scales topic replicas across brokers or racks to rebalance storage and adjust fault tolerance.
- Agentless Plugin Systems - Extends Kafka management capabilities through hot-pluggable modules without modifying broker configurations or restarting clusters.
- Cluster Health Monitoring - Displays real-time metrics and health status of Kafka clusters through a graphical interface without agent deployment.
- Kafka Cluster Health Monitors - Views real-time metrics and health status of Kafka clusters without modifying existing broker deployments.
- Automatic Cluster Discoverers - Discovers and registers existing Kafka clusters of any version with zero configuration changes to the brokers.
- Cluster Onboarding Tools - Discovers and registers existing Kafka clusters of any version with zero configuration changes to the brokers.
- Agentless Metric Collectors - Collects and aggregates broker, topic, and consumer metrics via non-intrusive polling without requiring agent deployment.
