# shopify/sarama

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/shopify-sarama).**

12,490 stars · 1,843 forks · Go · MIT

## Links

- GitHub: https://github.com/Shopify/sarama
- awesome-repositories: https://awesome-repositories.com/repository/shopify-sarama.md

## Description

Sarama is a Go client library for producing and consuming messages from Apache Kafka clusters. It provides dedicated interfaces for a message producer to send typed data packets and a message consumer to read and process continuous data streams from Kafka topics.

The library includes a mocking framework and simulation layer that mimics Kafka broker behavior. These tools allow for the testing of client applications and messaging logic without requiring a live cluster.

The project handles cluster connectivity and integration through the implementation of the Kafka binary protocol over TCP. It covers the management of cluster metadata, partition state, and asynchronous event production.

## Tags

### Data & Databases

- [Kafka Client Libraries](https://awesome-repositories.com/f/data-databases/kafka-client-libraries.md) — Provides a comprehensive Go library for producing and consuming messages from Apache Kafka clusters.
- [Partition Offset Management](https://awesome-repositories.com/f/data-databases/partition-offset-management.md) — Tracks offset positions and partition assignments locally to manage data consumption and ensure ordered message processing.
- [Message Broker Connectivity](https://awesome-repositories.com/f/data-databases/redis-clients/cluster-connectivity/message-broker-connectivity.md) — Establishes communication between an application and a distributed Kafka cluster to exchange event streams. ([source](https://github.com/shopify/sarama#readme))
- [Cluster Topology Caching](https://awesome-repositories.com/f/data-databases/cluster-topology-caching.md) — Maintains a local cache of broker and partition leader maps to route requests efficiently without repeated network lookups.
- [Message Batching](https://awesome-repositories.com/f/data-databases/data-buffering/message-batching.md) — Implements internal buffering to collect multiple outgoing messages, reducing network overhead and increasing producer throughput.
- [Asynchronous Producer Pipelines](https://awesome-repositories.com/f/data-databases/shared-memory-data-exchange/concurrent-data-pipelines/asynchronous-producer-pipelines.md) — Utilizes Go channels to asynchronously dispatch messages to different partition leaders across concurrent routines.

### Part of an Awesome List

- [Distributed Messaging](https://awesome-repositories.com/f/awesome-lists/data/distributed-messaging.md) — Provides a typed client for sending data packets to a distributed message cluster for asynchronous communication. ([source](https://github.com/shopify/sarama#readme))
- [Go Libraries](https://awesome-repositories.com/f/awesome-lists/more/go-libraries.md) — Implemented as a library for the Go programming language to interact with Kafka clusters.
- [Messaging Systems](https://awesome-repositories.com/f/awesome-lists/devops/messaging-systems.md) — Library for interacting with Apache Kafka.

### DevOps & Infrastructure

- [Distributed Messaging](https://awesome-repositories.com/f/devops-infrastructure/distributed-messaging.md) — Provides capabilities for reading and processing continuous data streams from a distributed Kafka cluster.

### Networking & Communication

- [Binary Protocols](https://awesome-repositories.com/f/networking-communication/binary-protocols.md) — Implements the Kafka wire protocol by encoding and decoding binary messages over raw TCP sockets.
- [Message Broker Consumers](https://awesome-repositories.com/f/networking-communication/message-broker-consumers.md) — Implements a streaming interface for reading and processing continuous data streams from Kafka topics.
- [Message Broker Producers](https://awesome-repositories.com/f/networking-communication/message-broker-producers.md) — Provides a client implementation for sending typed data packets to a distributed Kafka message cluster.
- [Topic Message Listeners](https://awesome-repositories.com/f/networking-communication/messaging-api-integrations/topic-message-listeners.md) — Provides mechanisms for reading and processing continuous data streams from Kafka topics to trigger application logic. ([source](https://github.com/shopify/sarama#readme))

### Software Engineering & Architecture

- [Asynchronous Message Production](https://awesome-repositories.com/f/software-engineering-architecture/asynchronous-event-dispatchers/asynchronous-message-production.md) — Enables decoupled communication between microservices by sending data packets asynchronously to a Kafka cluster.

### Testing & Quality Assurance

- [Dependency Mocking](https://awesome-repositories.com/f/testing-quality-assurance/dependency-mocking.md) — Uses Go interfaces to decouple the client logic from the network layer for simulating broker behavior during tests.
- [Cluster Integration Testing](https://awesome-repositories.com/f/testing-quality-assurance/in-process-server-testing/cluster-integration-testing.md) — Simulates cluster behavior with mock components to validate messaging logic without requiring a live broker.
- [Mocking Frameworks](https://awesome-repositories.com/f/testing-quality-assurance/mocking-frameworks.md) — Provides a set of tools and simulation layers to mimic Kafka broker behavior for testing client applications.