dpkp/kafka-python

Features

• Kafka Client Libraries - A pure-Python library implementing the Kafka wire protocol for producing and consuming messages.
• Database Wire Protocol Implementations - Implements the Kafka wire protocol directly in Python without native bindings or JVM dependencies.
• Consumer Group Clients - Reads messages from Kafka topics with automatic broker failure handling and partition rebalancing across a consumer group.
• Consumer Libraries - Provides a Python consumer library with consumer group coordination and automatic offset management.
• Go Consumer Libraries - Reads records from Kafka topics with automatic partition assignment and offset management.
• Producer Libraries - Ships a Python producer library with asynchronous batching, compression, and transactional semantics.
• Asynchronous Producer Pipelines - Buffers outgoing records in memory and sends them in batches on a background thread for high throughput.
• Kafka Broker Connections - Implements the Kafka wire protocol directly to connect to brokers from version 0.8.0 through 4.3 and later.
• Kafka Event Publishing - Publishes records to Kafka topics with configurable partitioning and acknowledgment guarantees.
• Message Broker Consumers - Reads messages from Kafka topics, automatically handling broker failures and partition rebalancing.
• Consumer Group Coordinators - Coordinates partition assignment across consumer instances using the Kafka group coordinator protocol for load-balanced consumption.
• Consumer Group Managers - Distributes Kafka topic partitions across multiple consumer instances with automatic rebalancing and offset management.
• Message Stream Consumer Groups - Reads messages from Kafka topics as an iterable with consumer group and deserialization support.
• Topic Producers - Provides an asynchronous producer that sends keyed or keyless messages to Kafka topics with batching and compression.
• Delivery Guarantees - Provides configurable acknowledgment criteria and automatic retries for balancing throughput against durability.
• Kafka Cluster Administration - Provides a programmatic admin client and CLI for creating, describing, and managing Kafka topics and cluster metadata.
• Pluggable Serializers - Applies user-supplied serializer and deserializer functions to message keys and values before network I/O.
• Transactional Producer Semantics - Groups produce operations into atomic transactions using Kafka's idempotent producer and transaction coordinator.
• Transactional Message Batches - Groups multiple message sends into atomic transactions that commit or abort as a single unit.
• Transactional Message Producers - Implements a transactional producer that groups message sends into atomic commits or aborts across partitions.
• Transactional Message Writes - Groups multiple message sends into atomic transactions that can be committed or aborted as a unit.
• Transactional Message Deliveries - Delivers messages to multiple partitions and topics as a single atomic unit using the transactional producer.
• Admin Clients - Ships a Python admin client for programmatic management of Kafka topics and cluster metadata.
• CLI Client Tools - Starts a consumer process from the command line that reads and prints messages from a specified Kafka topic.
• Transactional Producers - Implements transactional producer semantics for atomic writes across multiple Kafka partitions.
• Transactional Message Consumers - Consumes only committed messages from transactional topics, excluding uncommitted or aborted ones.
• Rack-Aware Fetch Routings - Directs fetch requests to a replica in the same rack as the consumer to reduce network traffic between data centers.
• CLI Client Tools - Ships command-line tools for producing, consuming, and administering Kafka topics without writing code.
• Manual Partition Assignments - Allows direct assignment of specific topic partitions to a consumer, bypassing automatic group coordination.
• Consumer Group Offset Managers - Commits the current position for a consumer group so consumption can resume from that point after a restart.
• Static Consumer Memberships - Assigns a unique instance ID to prevent group rebalances during transient unavailability like process restarts.
• CLI Client Tools - Starts a producer process from the command line that reads input and sends messages to a specified Kafka topic.
• Kafka Record Compressors - Compresses Kafka records with the LZ4 algorithm to reduce network and storage usage.
• Kafka Record Compressors - Compresses Kafka records with the Snappy algorithm to reduce network and storage usage.
• Message Compression Codecs - Compresses Kafka messages using gzip, LZ4, Snappy, or Zstandard codecs to reduce network bandwidth.
• Pluggable Compression Codecs - Wraps gzip, Snappy, LZ4, and Zstandard compression libraries behind a unified pluggable codec interface.
• Kafka Record Compressors - Compresses Kafka records with the Zstandard algorithm to reduce network and storage usage.
• Transactional Record Senders - Sends records across multiple partitions and topics as atomic transactions for all-or-nothing delivery.
• CLI Client Tools - Operates Kafka clients from the command line using module entry points for consumer, producer, and admin tasks.
• Idempotent Record Producers - Provides an idempotent producer mode that prevents duplicate records from retries.
• Idempotent Message Producers - Ships an idempotent producer mode that prevents duplicate records from retries.
• Transactional Message Batches - Groups produce operations into atomic transactions that can be committed or aborted as a unit.
• CLI Client Tools - Provides CLI tools for creating topics, describing cluster state, and running producer and consumer operations.
• Command-Line Tools - Provides command-line tools for producing, consuming, and administering Kafka topics without writing code.
• Kafka Fetch Configurations - Configures how much data to fetch per request and how long to wait for data to accumulate before returning.
• Database Clients - Client for Apache Kafka.
• Database Drivers - Apache Kafka client.

Star history