Awesome GitHub RepositoriesKafka Client Integrations

Sarama is an Apache Kafka Go client library that provides native support for the Kafka protocol. It includes a protocol client for managing offsets and timestamps, a producer implementation for sending messages, and a consumer group coordinator to balance workloads across multiple instances. The library enables high throughput data streaming through concurrent message production and maintains strict partition ordering during network retries. It supports secure communication with Kafka brokers using certificate-based encryption to protect data traffic. The project covers a broad range of dist

Connect is a Kafka data integration platform and stream processing engine used to build declarative pipelines that move and transform messages between Kafka topics and external sources. It functions as a Kafka Connect framework and a change data capture tool, streaming real-time database modifications to synchronize data across distributed environments. The project differentiates itself through a dedicated mapping language for mutating and reshaping message payloads and the ability to execute custom processing logic within a sandboxed WebAssembly runtime. It also provides an observability pip

Hyperf is a high-performance PHP coroutine framework designed for building microservices and middleware. It utilizes non-blocking coroutines to handle high concurrency and low-latency request processing, providing a foundation for scalable distributed systems. The framework is distinguished by an aspect-oriented programming based dependency injector that enables pluggable components and meta-programming. It includes a coroutine-optimized object-relational mapper with integrated model caching and an orchestration toolkit for microservice governance, featuring service discovery, circuit breaker

aws-sam-local ist ein Docker-basierter Emulator, der zum Ausführen und Debuggen von AWS Lambda-Funktionen und API-Gateway-Konfigurationen auf einem lokalen Rechner verwendet wird. Er bietet eine lokale Laufzeitumgebung, die die AWS Lambda-Runtime nachahmt und die Verifizierung von Funktionscode sowie serverlosen Anwendungs-Mocks vor der Cloud-Bereitstellung ermöglicht. Das Projekt nutzt einen Runtime-Orchestrator zur Verwaltung ephemerer Container, die die isolierte Ausführungsumgebung von Cloud-Funktionen replizieren. Es enthält einen Serverless-API-Gateway-Simulator und Event-Mocking-Tools, um Payloads aus Quellen wie Kafka und anderen Cloud-Triggern zu synthetisieren. Das Tool unterstützt den breiteren Entwicklungszyklus für das Serverless Application Model, indem es lokales Prototyping von serverlosen APIs und die Simulation verschiedener Cloud-Event-Quellen ermöglicht. Es bietet eine Debugging-Umgebung zur Identifizierung von Fehlern in serverlosen Funktionen mit voller Kontrolle über den Ausführungsstatus.

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

kafka-python is a pure-Python client library for Apache Kafka that implements the Kafka wire protocol directly, without any native bindings or JVM dependencies. It provides the core capabilities of a Kafka client: producing messages to topics, consuming records from topics, and administering cluster resources such as topics and partitions, all through a Pythonic API or command-line tools. The library distinguishes itself through its comprehensive support for advanced Kafka features. It includes an asynchronous producer with background batching for throughput, a consumer group rebalance protoc

Strimzi is a Kubernetes operator that automates the deployment, management, and lifecycle of Apache Kafka clusters on Kubernetes or OpenShift. It uses custom resource definitions and declarative YAML configuration to define Kafka cluster topology, broker placement, and security settings, with operator-based controllers that reconcile the desired state with the actual cluster state. The operator handles rolling updates during cluster upgrades or configuration changes to maintain availability and data integrity, and supports rack-aware broker scheduling across Kubernetes nodes and availability

FastStream is an asynchronous Python framework designed for building event-driven microservices. It provides a unified abstraction layer for interacting with various message brokers, enabling developers to manage event production and consumption through a consistent interface while maintaining access to native provider-specific features. The framework centers on a decorator-based routing model that binds application logic directly to broker topics, supported by a built-in dependency injection container that resolves resources at runtime. The framework distinguishes itself through its deep int

KafkaJS ist ein reiner JavaScript-Client für Apache Kafka, der die notwendigen Tools bereitstellt, um Nachrichten aus einem Kafka-Cluster zu produzieren und zu konsumieren, ohne native Abhängigkeiten oder externe Addons zu erfordern. Er fungiert als umfassende Integrationsbibliothek für Node.js-Anwendungen, um an verteilter Nachrichtenverarbeitung und Echtzeit-Event-Streaming teilzunehmen. Das Projekt zeichnet sich durch seine native Implementierung des Kafka-Wire-Protokolls aus, wodurch C++-Abhängigkeiten vermieden werden. Es verfügt über einen Security-Client, der SSL-, TLS- und SASL-Authentifizierung unterstützt, sowie über transaktionale Funktionen, die atomares Nachrichten-Senden und verknüpfte Offset-Commitments ermöglichen, um Exactly-Once-Processing sicherzustellen. Die Bibliothek deckt ein breites Spektrum an operativen Bereichen ab, einschließlich vollständiger Cluster-Administration zur Verwaltung von Topics und Consumer-Groups, fortgeschrittenem Partition-Routing und Zuweisungsstrategien sowie umfassender Telemetrie durch event-gesteuertes Monitoring. Sie implementiert zudem Netzwerk-Zuverlässigkeitsmuster wie Exponential-Backoff-Retries und Rack-Aware-Data-Fetching, um die Latenz zu optimieren.

AWS Powertools for Python is a utility framework designed for building production-ready Python functions on AWS Lambda. It provides a comprehensive suite of tools for observability, event parsing, routing, and idempotency management to streamline the development of serverless applications. The project distinguishes itself through specialized capabilities for event-driven architectures and AI agent orchestration. It enables the implementation of AI agents by exposing functions as tools via OpenAPI schemas and managing conversation states. Additionally, it features an idempotency library that p

franz-go is a low-level Go client library and wire protocol implementation for producing, consuming, and administering Kafka clusters. It functions as a zero-allocation network driver that utilizes a direct TCP communication layer to handle requests and responses. The project integrates a schema registry client for encoding and decoding structured data. It provides a programmatic interface for cluster administration, including the management of topics, access control lists, and broker configurations. The library covers data consumption through consumer groups, message production with transac