dpkp/kafka-python
Kafka Python
kafka-python is a pure-Python client library for Apache Kafka that implements the Kafka wire protocol directly, without any native bindings or JVM dependencies. It provides the core capabilities of a Kafka client: producing messages to topics, consuming records from topics, and administering cluster resources such as topics and partitions, all through a Pythonic API or command-line tools.
The library distinguishes itself through its comprehensive support for advanced Kafka features. It includes an asynchronous producer with background batching for throughput, a consumer group rebalance protocol for load-balanced consumption across multiple instances, and transactional producer semantics for atomic writes across partitions. It also offers rack-aware fetch routing to minimize cross-datacenter traffic, pluggable serialization for message keys and values, and hardware-accelerated CRC32C checksums via an optimized C extension. Compression is supported through gzip, Snappy, LZ4, and Zstandard algorithms.
Beyond the core client, kafka-python provides a full set of command-line tools for producing, consuming, and administering topics without writing code. It includes an admin client for programmatic cluster management, and exposes performance metrics from producer and consumer instances for monitoring throughput and latency. The library supports Python versions 3.8 through 3.14 and connects to Kafka broker versions from 0.8.0 through 4.3 and later.
Features
- Kafka Client Libraries - A pure-Python library implementing the Kafka wire protocol for producing and consuming messages.
- Database Wire Protocol Implementations - Implements the Kafka wire protocol directly in Python without native bindings or JVM dependencies.
- Consumer Group Clients - Reads messages from Kafka topics with automatic broker failure handling and partition rebalancing across a consumer group.
- Consumer Libraries - Provides a Python consumer library with consumer group coordination and automatic offset management.
- Go Consumer Libraries - Reads records from Kafka topics with automatic partition assignment and offset management.
- Producer Libraries - Ships a Python producer library with asynchronous batching, compression, and transactional semantics.
- Asynchronous Producer Pipelines - Buffers outgoing records in memory and sends them in batches on a background thread for high throughput.
- Kafka Broker Connections - Implements the Kafka wire protocol directly to connect to brokers from version 0.8.0 through 4.3 and later.
- Kafka Event Publishing - Publishes records to Kafka topics with configurable partitioning and acknowledgment guarantees.
- Message Broker Consumers - Reads messages from Kafka topics, automatically handling broker failures and partition rebalancing.
- Consumer Group Coordinators - Coordinates partition assignment across consumer instances using the Kafka group coordinator protocol for load-balanced consumption.
- Consumer Group Managers - Distributes Kafka topic partitions across multiple consumer instances with automatic rebalancing and offset management.
- Message Stream Consumer Groups - Reads messages from Kafka topics as an iterable with consumer group and deserialization support.
- Topic Producers - Provides an asynchronous producer that sends keyed or keyless messages to Kafka topics with batching and compression.
- Delivery Guarantees - Provides configurable acknowledgment criteria and automatic retries for balancing throughput against durability.
- Kafka Cluster Administration - Provides a programmatic admin client and CLI for creating, describing, and managing Kafka topics and cluster metadata.
- Pluggable Serializers - Applies user-supplied serializer and deserializer functions to message keys and values before network I/O.
- Transactional Producer Semantics - Groups produce operations into atomic transactions using Kafka's idempotent producer and transaction coordinator.
- Transactional Message Batches - Groups multiple message sends into atomic transactions that commit or abort as a single unit.
- Transactional Message Producers - Implements a transactional producer that groups message sends into atomic commits or aborts across partitions.
- Transactional Message Writes - Groups multiple message sends into atomic transactions that can be committed or aborted as a unit.
- Transactional Message Deliveries - Delivers messages to multiple partitions and topics as a single atomic unit using the transactional producer.
- Admin Clients - Ships a Python admin client for programmatic management of Kafka topics and cluster metadata.
- CLI Client Tools - Starts a consumer process from the command line that reads and prints messages from a specified Kafka topic.
- Transactional Producers - Implements transactional producer semantics for atomic writes across multiple Kafka partitions.
- Transactional Message Consumers - Consumes only committed messages from transactional topics, excluding uncommitted or aborted ones.
- Rack-Aware Fetch Routings - Directs fetch requests to a replica in the same rack as the consumer to reduce network traffic between data centers.
- CLI Client Tools - Ships command-line tools for producing, consuming, and administering Kafka topics without writing code.
- Manual Partition Assignments - Allows direct assignment of specific topic partitions to a consumer, bypassing automatic group coordination.
- Consumer Group Offset Managers - Commits the current position for a consumer group so consumption can resume from that point after a restart.
- Static Consumer Memberships - Assigns a unique instance ID to prevent group rebalances during transient unavailability like process restarts.
- CLI Client Tools - Starts a producer process from the command line that reads input and sends messages to a specified Kafka topic.
- Kafka Record Compressors - Compresses Kafka records with the LZ4 algorithm to reduce network and storage usage.
- Kafka Record Compressors - Compresses Kafka records with the Snappy algorithm to reduce network and storage usage.
- Message Compression Codecs - Compresses Kafka messages using gzip, LZ4, Snappy, or Zstandard codecs to reduce network bandwidth.
- Pluggable Compression Codecs - Wraps gzip, Snappy, LZ4, and Zstandard compression libraries behind a unified pluggable codec interface.
- Kafka Record Compressors - Compresses Kafka records with the Zstandard algorithm to reduce network and storage usage.
- Transactional Record Senders - Sends records across multiple partitions and topics as atomic transactions for all-or-nothing delivery.
- CLI Client Tools - Operates Kafka clients from the command line using module entry points for consumer, producer, and admin tasks.
- Idempotent Record Producers - Provides an idempotent producer mode that prevents duplicate records from retries.
- Idempotent Message Producers - Ships an idempotent producer mode that prevents duplicate records from retries.
- Transactional Message Batches - Groups produce operations into atomic transactions that can be committed or aborted as a unit.
- CLI Client Tools - Provides CLI tools for creating topics, describing cluster state, and running producer and consumer operations.
- Command-Line Tools - Provides command-line tools for producing, consuming, and administering Kafka topics without writing code.
- Kafka Fetch Configurations - Configures how much data to fetch per request and how long to wait for data to accumulate before returning.
- Database Clients - Client for Apache Kafka.
- Database Drivers - Apache Kafka client.
