twmb/franz-go
Franz Go
franz-go is a low-level Go client library and wire protocol implementation for producing, consuming, and administering Kafka clusters. It functions as a zero-allocation network driver that utilizes a direct TCP communication layer to handle requests and responses.
The project integrates a schema registry client for encoding and decoding structured data. It provides a programmatic interface for cluster administration, including the management of topics, access control lists, and broker configurations.
The library covers data consumption through consumer groups, message production with transaction support, and secure communication via encrypted and authenticated connections. It also includes hooks for monitoring client performance, network latencies, and throughput.
Features
- High-Throughput Data Streaming - Implements a high-throughput Kafka client in Go using low-allocation drivers and request pipelining.
- Kafka Client Libraries - Implements a low-level Go client library for producing, consuming, and administering Kafka clusters.
- Kafka Client Integrations - Provides Go client integration for producing messages to Kafka with support for transactions and compression.
- Kafka Message Production - Sends data to Kafka brokers with support for transactions and compression to ensure reliable delivery.
- Kafka Stream Exporters - Implements efficient data consumption from Kafka brokers using consumer groups for parallel processing.
- Kafka Protocol Implementations - Provides a low-level implementation of the Kafka wire protocol over direct TCP sockets.
- Message Stream Consumer Groups - Reads data from Kafka topics using consumer groups for efficient parallel processing across workers.
- Kafka Cluster Administration - Provides a programmatic interface for administering Kafka brokers, topics, and cluster configurations.
- Schema Registries - Integrates with a schema registry to manage typed metadata and consistent data formats across services.
- Schema Registry Clients - Includes a schema registry client for encoding and decoding structured data using specified schemas.
- Network Drivers - Functions as a zero-allocation network driver using buffer pools and pipelining to minimize memory overhead.
- Kafka Security Integrations - Implements authentication and encryption protocols specifically for securing Kafka client connections.
- Pluggable Authentication Handlers - Provides a pluggable architecture for authentication handlers to decouple security schemes from protocol logic.
- Secure Connection Managers - Establishes encrypted and authenticated communication channels using secure connection managers.
- Security Dialers - Decouples connection establishment from protocol logic using wrappers for TLS and SASL authentication.
- Asynchronous Request Pipelines - Implements asynchronous request pipelining to increase throughput by sending multiple requests without waiting for immediate responses.
- Byte Buffer Pooling - Utilizes a pool of reusable byte buffers to reduce garbage collection overhead during network I/O operations.
- Data Encoders and Decoders - Handles the encoding and decoding of structured data by interacting with a schema registry.
- Interface-Based Schema Resolution - Abstracts serialization logic through generic interfaces to support multiple schema registry versions and encoding formats.
- Zero-Allocation Buffer Encoders - Implements zero-allocation serialization by reusing buffers and encoding directly into byte slices.
