# apache/iggy

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/apache-iggy).**

4,382 stars · 348 forks · Rust · Apache-2.0

## Links

- GitHub: https://github.com/apache/iggy
- Homepage: https://iggy.apache.org
- awesome-repositories: https://awesome-repositories.com/repository/apache-iggy.md

## Topics

`apache` `http` `iggy` `messaging` `quic` `rust` `streaming` `tcp`

## Description

Iggy is a distributed message streaming platform and multi-protocol message broker that functions as a persistent distributed log store. It provides infrastructure for publishing and consuming binary messages using an append-only log, ensuring high availability and data consistency across nodes through Viewstamped Replication.

The platform is distinguished by its specialized LLM streaming infrastructure, which uses a server protocol to connect large language models to streaming data and system controls. This includes standardized protocols for context management and data bridging via HTTP or standard input-output.

The system covers a broad set of capabilities including data pipeline orchestration with modular source and sink plugins, consumer group coordination for horizontal scaling, and multi-protocol transport support across TCP, QUIC, HTTP, and WebSocket. It also incorporates security primitives such as AES-256-GCM encryption for data at rest and in transit, and provides observability via Prometheus metrics, OpenTelemetry tracing, and an operational web dashboard.

The server can be deployed using container images and orchestrated through Kubernetes.

## Tags

### Data & Databases

- [Append-Only Message Logs](https://awesome-repositories.com/f/data-databases/append-only-message-logs.md) — Implements a high-throughput persistent append-only log for reading and writing raw binary data. ([source](https://cdn.jsdelivr.net/gh/apache/iggy@master/README.md))
- [Append-Only Storage Engines](https://awesome-repositories.com/f/data-databases/append-only-storage-engines.md) — Persists data as immutable sequences of records to enable multiple consumers to read and replay data. ([source](https://iggy.apache.org/docs/introduction/architecture))
- [Distributed Data Replication](https://awesome-repositories.com/f/data-databases/client-server-data-synchronization/core-data-sync-servers/distributed-data-replication.md) — Synchronizes and replicates data across multiple cluster nodes using Viewstamped Replication for high availability. ([source](https://iggy.apache.org/docs/server/configuration))
- [Consensus Replication Protocols](https://awesome-repositories.com/f/data-databases/consensus-replication-protocols.md) — Ensures high availability and data consistency across a cluster using the Viewstamped Replication consensus protocol.
- [Consumer Offset Trackers](https://awesome-repositories.com/f/data-databases/consumer-offset-trackers.md) — Tracks the read progress of consumers using sequence positions (offsets) within data streams.
- [Persistence & Durability](https://awesome-repositories.com/f/data-databases/data-engineering-infrastructure/data-persistence-storage/persistence-durability.md) — Ensures data persistence during crashes through filesystem synchronization and configurable background flush intervals. ([source](https://iggy.apache.org/docs/server/configuration))
- [Horizontal Scaling](https://awesome-repositories.com/f/data-databases/horizontal-scaling.md) — Distributes data across multiple partitions to increase parallel processing and enable horizontal scaling. ([source](https://iggy.apache.org/docs/introduction/architecture))
- [Consumer Group Scaling](https://awesome-repositories.com/f/data-databases/horizontal-scaling/consumer-group-scaling.md) — Enables horizontal scaling of message consumption through coordinated consumer groups and partition rebalancing. ([source](https://cdn.jsdelivr.net/gh/apache/iggy@master/README.md))
- [Data Compression Algorithms](https://awesome-repositories.com/f/data-databases/data-compression-algorithms.md) — Implements multiple compression algorithms including gzip, lz4, and zstd to optimize on-disk storage capacity. ([source](https://iggy.apache.org/docs/introduction/architecture))
- [Partition Index Caching](https://awesome-repositories.com/f/data-databases/data-engineering-infrastructure/caching-performance/caching-strategies/hybrid-memory-disk-caches/partition-index-caching.md) — Manages how partition indexes are stored in memory or read from disk to balance memory usage and access speed. ([source](https://iggy.apache.org/docs/introduction/architecture))
- [Topic Management](https://awesome-repositories.com/f/data-databases/data-engineering-infrastructure/data-persistence-storage/data-storage/topic-management.md) — Allows the creation and organization of named channels to distribute and partition streaming data. ([source](https://iggy.apache.org/docs/ai/mcp))
- [Custom Connector Development](https://awesome-repositories.com/f/data-databases/data-i-o/custom-connector-development.md) — Enables the development of custom input and output adapters to integrate the platform with external data sources. ([source](https://iggy.apache.org/docs/connectors/introduction/))
- [Data Pipeline Orchestration](https://awesome-repositories.com/f/data-databases/data-pipeline-orchestration.md) — Orchestrates data movement between external sources and systems using modular plugins. ([source](https://iggy.apache.org/docs/introduction/about/))
- [Data Retention Policies](https://awesome-repositories.com/f/data-databases/data-retention-policies.md) — Provides automated data deletion based on configurable time or size policies to manage disk usage. ([source](https://cdn.jsdelivr.net/gh/apache/iggy@master/README.md))
- [External Data Ingestion](https://awesome-repositories.com/f/data-databases/external-data-ingestion.md) — Polls data from external systems and pushes it into streams via modular source plugins. ([source](https://iggy.apache.org/docs/connectors/introduction/))
- [Stream Data Exporters](https://awesome-repositories.com/f/data-databases/external-data-integrations/stream-data-exporters.md) — Forwards messages from streams to external destinations using offset-tracking plugins. ([source](https://iggy.apache.org/docs/connectors/introduction))
- [External System Integrations](https://awesome-repositories.com/f/data-databases/external-system-integrations.md) — Bridges streams with external databases and search engines using a plugin system with built-in transformations. ([source](https://iggy.apache.org/docs/))
- [Data Stream Management](https://awesome-repositories.com/f/data-databases/real-time-data-streaming/data-stream-management.md) — Manages the full lifecycle of append-only streams, including updates, deletions, and purging of partitions. ([source](https://iggy.apache.org/docs/ai/mcp/))
- [Topic Partition Mapping](https://awesome-repositories.com/f/data-databases/topic-partition-mapping.md) — Controls how logical topics are partitioned and mapped to distribute message loads across the cluster. ([source](https://iggy.apache.org/docs/ai/mcp))

### DevOps & Infrastructure

- [Distributed Log Stores](https://awesome-repositories.com/f/devops-infrastructure/single-node-deployment/single-process-servers/multi-model-server-architectures/peer-to-peer-storage-models/distributed-data-stores/distributed-log-stores.md) — Implements a persistent distributed log store using Viewstamped Replication for high availability and data consistency.
- [Partition Distribution Strategies](https://awesome-repositories.com/f/devops-infrastructure/distributed-messaging/partition-distribution-strategies.md) — Distributes data across partitions using round-robin or key-based hashing to optimize message polling and sending. ([source](https://iggy.apache.org/docs/sdk/introduction))
- [Message Consumption](https://awesome-repositories.com/f/devops-infrastructure/queue-management/message-queues/message-consumption.md) — Provides mechanisms for producing and consuming messages with support for partitioning and offset management. ([source](https://iggy.apache.org/docs/ai/mcp))
- [Control and Data Plane Separation](https://awesome-repositories.com/f/devops-infrastructure/control-planes/control-and-data-plane-separation.md) — Architecturally separates replication into a control plane for metadata and a data plane for high-speed streaming.

### Artificial Intelligence & ML

- [LLM Streaming Infrastructure](https://awesome-repositories.com/f/artificial-intelligence-ml/llm-streaming-infrastructure.md) — Exposes a specialized server protocol that connects large language models to streaming data and operational system controls.
- [Model Context Protocol Integrations](https://awesome-repositories.com/f/artificial-intelligence-ml/model-context-protocol-integrations.md) — Provides standardized interfaces to link large language models to streaming data and system infrastructure. ([source](https://iggy.apache.org/docs/introduction/about/))
- [LLM Application Infrastructure](https://awesome-repositories.com/f/artificial-intelligence-ml/llm-application-infrastructure.md) — Provides backend infrastructure to connect large language models to streaming data through standard network transports. ([source](https://iggy.apache.org/docs/ai/mcp))
- [Stream Operation Controllers](https://awesome-repositories.com/f/artificial-intelligence-ml/llm-stream-processing/stream-operation-controllers.md) — Exposes a specialized server protocol that enables large language models to control message streaming operations. ([source](https://iggy.apache.org/docs/))

### Part of an Awesome List

- [Data Ingestion Pipelines](https://awesome-repositories.com/f/awesome-lists/data/data-ingestion-pipelines.md) — Provides a modular system for ingesting and forwarding data between external sources and streams using plugins.
- [Stream Administration](https://awesome-repositories.com/f/awesome-lists/data/storage-and-backups/storage-administration/stream-administration.md) — Provides administrative client methods to create and configure streams, topics, and partitions. ([source](https://iggy.apache.org/docs/introduction/getting-started))

### Networking & Communication

- [Binary Protocols](https://awesome-repositories.com/f/networking-communication/binary-protocols.md) — Uses a custom binary protocol for efficient data transmission across TCP, QUIC, WebSocket, and HTTP transports. ([source](https://iggy.apache.org/docs/))
- [Consumer Group Coordinators](https://awesome-repositories.com/f/networking-communication/message-broker-consumers/consumer-group-coordinators.md) — Coordinates multiple consumers in groups to ensure partition-level exclusivity and horizontal scaling. ([source](https://iggy.apache.org/docs/ai/mcp))
- [Message Brokers](https://awesome-repositories.com/f/networking-communication/message-brokers.md) — Functions as a multi-protocol message broker supporting data transmission via TCP, QUIC, HTTP, and WebSocket.
- [Stream Offset Tracking](https://awesome-repositories.com/f/networking-communication/message-stream-handlers/message-stream-consumer-groups/stream-offset-tracking.md) — Manages message delivery by tracking consumer and server offsets within a stream. ([source](https://iggy.apache.org/docs/introduction/getting-started))
- [Stream Topic Publishing](https://awesome-repositories.com/f/networking-communication/messaging-notifications/stream-topic-publishing.md) — Sends records to specific streams and partitions using optimized transport protocols for high-throughput data ingestion. ([source](https://iggy.apache.org/docs/introduction/getting-started))
- [Multi-Format Message Serialization](https://awesome-repositories.com/f/networking-communication/multi-format-message-serialization.md) — Supports multiple serialization formats including JSON, Protocol Buffers, and FlatBuffers for efficient binary storage. ([source](https://iggy.apache.org/docs/connectors/introduction))
- [Multi-Protocol Transport Layers](https://awesome-repositories.com/f/networking-communication/multi-protocol-transport-layers.md) — Routes binary messages via a multi-protocol transport layer supporting TCP, QUIC, HTTP, and WebSockets.
- [Multi-Transport Support](https://awesome-repositories.com/f/networking-communication/network-transport-protocols/multi-transport-support.md) — Supports multiple communication layers including TCP, QUIC, HTTP, and WebSockets to balance throughput and compatibility. ([source](https://iggy.apache.org/docs/server/configuration))
- [Streaming Data Integrations](https://awesome-repositories.com/f/networking-communication/streaming-data-integrations.md) — Encodes and decodes stream data using various binary and text formats. ([source](https://iggy.apache.org/docs/connectors/introduction/))
- [Cross-Shard Connection Migration](https://awesome-repositories.com/f/networking-communication/cross-shard-connection-migration.md) — Moves client connections to the shard owning the target partition to remove the overhead of cross-shard messaging. ([source](https://iggy.apache.org/docs/introduction/architecture))
- [AI Messaging Bridges](https://awesome-repositories.com/f/networking-communication/message-bridges/ai-messaging-bridges.md) — Provides integration layers that bridge message streaming infrastructure to large language models via HTTP or standard IO. ([source](https://iggy.apache.org/docs/ai/mcp/))
- [Stream Payload Transformations](https://awesome-repositories.com/f/networking-communication/stream-payload-transformations.md) — Modifies message fields by adding, deleting, or updating values before data enters or leaves a stream. ([source](https://iggy.apache.org/docs/connectors/introduction))

### Security & Cryptography

- [Permission-Based Access Control](https://awesome-repositories.com/f/security-cryptography/permission-based-access-control.md) — Enforces a permission-based access control model to restrict administrative and data-polling actions. ([source](https://iggy.apache.org/docs/introduction/getting-started))
- [TLS Traffic Encryption](https://awesome-repositories.com/f/security-cryptography/tls-traffic-encryption.md) — Secures network traffic across TCP, QUIC, HTTP, and WebSocket protocols using TLS encryption. ([source](https://iggy.apache.org/docs/server/security/))
- [User Account Management](https://awesome-repositories.com/f/security-cryptography/user-account-management.md) — Provides tools to create and manage user accounts and assign platform permissions. ([source](https://iggy.apache.org/docs/web_ui/start/))
- [AES-GCM Data Encryptors](https://awesome-repositories.com/f/security-cryptography/privacy-data-protection/data-encryption/end-to-end-encryption/media-encryption/stream-encryption-and-decryption/aes-gcm-data-encryptors.md) — Provides AES-GCM encryption to secure sensitive message data at rest. ([source](https://iggy.apache.org/docs/introduction/about/))
- [Transit and At-Rest Encryption](https://awesome-repositories.com/f/security-cryptography/privacy-data-protection/data-encryption/end-to-end-encryption/transit-and-at-rest-encryption.md) — Protects message payloads and state commands on disk using managed encryption. ([source](https://iggy.apache.org/docs/server/security/))
- [AES-Encrypted Transports](https://awesome-repositories.com/f/security-cryptography/transport-layer-encryption/aes-encrypted-transports.md) — Secures message payloads using AES-256-GCM encryption at both the producer and storage levels.

### Software Engineering & Architecture

- [Control and Data Plane Separation](https://awesome-repositories.com/f/software-engineering-architecture/control-and-data-plane-separation.md) — Separates replication into a control plane for managing users and topics and a data plane for streaming messages. ([source](https://iggy.apache.org/docs/clustering/vsr/))
- [Thread-Per-Core Architectures](https://awesome-repositories.com/f/software-engineering-architecture/performance-reliability/performance-optimization-patterns/thread-per-core-architectures.md) — Uses a thread-per-core architecture to map shards to specific CPU cores, improving hardware utilization and throughput. ([source](https://iggy.apache.org/docs/server/configuration))
- [Replicated State Machines](https://awesome-repositories.com/f/software-engineering-architecture/state-machine-logic/replicated-state-machines.md) — Maintains identical state across cluster nodes by applying an ordered sequence of transactions via a consensus protocol. ([source](https://iggy.apache.org/docs/clustering/vsr/))
- [Data Pipeline Extensions](https://awesome-repositories.com/f/software-engineering-architecture/plugin-based-logic-extensions/transformation-logic-extensions/data-pipeline-extensions.md) — Allows creation of custom source, sink, or transformation logic by implementing specific traits for system integration. ([source](https://iggy.apache.org/docs/connectors/introduction))
- [Trait-Based Plugin Interfaces](https://awesome-repositories.com/f/software-engineering-architecture/software-architecture/architectural-patterns/plugin-module-systems/modular-plugin-architectures/plugin-based-architectures/plugin-based-architectures/trait-based-plugin-interfaces.md) — Implements a modular plugin system using trait-based interfaces to create custom source and sink connectors.

### Development Tools & Productivity

- [CLI Administration Tools](https://awesome-repositories.com/f/development-tools-productivity/cli-administration-tools.md) — Provides a command-line interface for administering streams, topics, partitions, and message flow. ([source](https://iggy.apache.org/docs/cli/start/))

### Operating Systems & Systems Programming

- [CPU Affinity Binding](https://awesome-repositories.com/f/operating-systems-systems-programming/cpu-affinity-binding.md) — Binds shards to specific CPU cores and detects hardware topology to maximize processing efficiency. ([source](https://iggy.apache.org/docs/introduction/architecture))

### System Administration & Monitoring

- [Administrative Dashboards](https://awesome-repositories.com/f/system-administration-monitoring/administrative-dashboards.md) — Provides a web-based administrative dashboard to manage streams, topics, and user permissions. ([source](https://iggy.apache.org/docs/))
- [CLI System Management](https://awesome-repositories.com/f/system-administration-monitoring/cli-system-management.md) — Offers a system management CLI featuring session-based login and named connection contexts. ([source](https://iggy.apache.org/docs/))
- [Observability Dashboards](https://awesome-repositories.com/f/system-administration-monitoring/observability-dashboards.md) — Aggregates system health metrics, OpenTelemetry traces, and Prometheus data into a unified observability dashboard.
- [System Health Monitoring](https://awesome-repositories.com/f/system-administration-monitoring/observability-instrumentation/system-health-monitoring.md) — Retrieves server statistics, cluster metadata, and client connection details to troubleshoot system performance. ([source](https://iggy.apache.org/docs/ai/mcp))
- [Web Dashboards](https://awesome-repositories.com/f/system-administration-monitoring/web-dashboards.md) — Ships a built-in web user interface to visualize server configuration, logs, and stream activity. ([source](https://iggy.apache.org/docs/introduction/about/))