Iggy

Iggy is a distributed message streaming platform and multi-protocol message broker that functions as a persistent distributed log store. It provides infrastructure for publishing and consuming binary messages using an append-only log, ensuring high availability and data consistency across nodes through Viewstamped Replication.

The platform is distinguished by its specialized LLM streaming infrastructure, which uses a server protocol to connect large language models to streaming data and system controls. This includes standardized protocols for context management and data bridging via HTTP or standard input-output.

The system covers a broad set of capabilities including data pipeline orchestration with modular source and sink plugins, consumer group coordination for horizontal scaling, and multi-protocol transport support across TCP, QUIC, HTTP, and WebSocket. It also incorporates security primitives such as AES-256-GCM encryption for data at rest and in transit, and provides observability via Prometheus metrics, OpenTelemetry tracing, and an operational web dashboard.

The server can be deployed using container images and orchestrated through Kubernetes.

Features

Append-Only Message Logs - Implements a high-throughput persistent append-only log for reading and writing raw binary data.
Distributed Log Stores - Implements a persistent distributed log store using Viewstamped Replication for high availability and data consistency.
LLM Streaming Infrastructure - Exposes a specialized server protocol that connects large language models to streaming data and operational system controls.
Model Context Protocol Integrations - Provides standardized interfaces to link large language models to streaming data and system infrastructure.
Data Ingestion Pipelines - Provides a modular system for ingesting and forwarding data between external sources and streams using plugins.
Append-Only Storage Engines - Persists data as immutable sequences of records to enable multiple consumers to read and replay data.
Distributed Data Replication - Synchronizes and replicates data across multiple cluster nodes using Viewstamped Replication for high availability.
Consensus Replication Protocols - Ensures high availability and data consistency across a cluster using the Viewstamped Replication consensus protocol.
Consumer Offset Trackers - Tracks the read progress of consumers using sequence positions (offsets) within data streams.
Persistence & Durability - Ensures data persistence during crashes through filesystem synchronization and configurable background flush intervals.
Horizontal Scaling - Distributes data across multiple partitions to increase parallel processing and enable horizontal scaling.
Consumer Group Scaling - Enables horizontal scaling of message consumption through coordinated consumer groups and partition rebalancing.
Partition Distribution Strategies - Distributes data across partitions using round-robin or key-based hashing to optimize message polling and sending.
Message Consumption - Provides mechanisms for producing and consuming messages with support for partitioning and offset management.
Binary Protocols - Uses a custom binary protocol for efficient data transmission across TCP, QUIC, WebSocket, and HTTP transports.
Consumer Group Coordinators - Coordinates multiple consumers in groups to ensure partition-level exclusivity and horizontal scaling.
Message Brokers - Functions as a multi-protocol message broker supporting data transmission via TCP, QUIC, HTTP, and WebSocket.
Stream Offset Tracking - Manages message delivery by tracking consumer and server offsets within a stream.
Stream Topic Publishing - Sends records to specific streams and partitions using optimized transport protocols for high-throughput data ingestion.
Multi-Format Message Serialization - Supports multiple serialization formats including JSON, Protocol Buffers, and FlatBuffers for efficient binary storage.
Multi-Protocol Transport Layers - Routes binary messages via a multi-protocol transport layer supporting TCP, QUIC, HTTP, and WebSockets.
Multi-Transport Support - Supports multiple communication layers including TCP, QUIC, HTTP, and WebSockets to balance throughput and compatibility.
Streaming Data Integrations - Encodes and decodes stream data using various binary and text formats.
Permission-Based Access Control - Enforces a permission-based access control model to restrict administrative and data-polling actions.
TLS Traffic Encryption - Secures network traffic across TCP, QUIC, HTTP, and WebSocket protocols using TLS encryption.
User Account Management - Provides tools to create and manage user accounts and assign platform permissions.
Control and Data Plane Separation - Separates replication into a control plane for managing users and topics and a data plane for streaming messages.
Thread-Per-Core Architectures - Uses a thread-per-core architecture to map shards to specific CPU cores, improving hardware utilization and throughput.
Replicated State Machines - Maintains identical state across cluster nodes by applying an ordered sequence of transactions via a consensus protocol.
LLM Application Infrastructure - Provides backend infrastructure to connect large language models to streaming data through standard network transports.
Stream Operation Controllers - Exposes a specialized server protocol that enables large language models to control message streaming operations.
Stream Administration - Provides administrative client methods to create and configure streams, topics, and partitions.
Data Compression Algorithms - Implements multiple compression algorithms including gzip, lz4, and zstd to optimize on-disk storage capacity.
Partition Index Caching - Manages how partition indexes are stored in memory or read from disk to balance memory usage and access speed.
Topic Management - Allows the creation and organization of named channels to distribute and partition streaming data.
Custom Connector Development - Enables the development of custom input and output adapters to integrate the platform with external data sources.
Data Pipeline Orchestration - Orchestrates data movement between external sources and systems using modular plugins.
Data Retention Policies - Provides automated data deletion based on configurable time or size policies to manage disk usage.
External Data Ingestion - Polls data from external systems and pushes it into streams via modular source plugins.
Stream Data Exporters - Forwards messages from streams to external destinations using offset-tracking plugins.
External System Integrations - Bridges streams with external databases and search engines using a plugin system with built-in transformations.
Data Stream Management - Manages the full lifecycle of append-only streams, including updates, deletions, and purging of partitions.
Topic Partition Mapping - Controls how logical topics are partitioned and mapped to distribute message loads across the cluster.
CLI Administration Tools - Provides a command-line interface for administering streams, topics, partitions, and message flow.
Control and Data Plane Separation - Architecturally separates replication into a control plane for metadata and a data plane for high-speed streaming.
Cross-Shard Connection Migration - Moves client connections to the shard owning the target partition to remove the overhead of cross-shard messaging.
AI Messaging Bridges - Provides integration layers that bridge message streaming infrastructure to large language models via HTTP or standard IO.
Stream Payload Transformations - Modifies message fields by adding, deleting, or updating values before data enters or leaves a stream.
CPU Affinity Binding - Binds shards to specific CPU cores and detects hardware topology to maximize processing efficiency.
AES-GCM Data Encryptors - Provides AES-GCM encryption to secure sensitive message data at rest.
Transit and At-Rest Encryption - Protects message payloads and state commands on disk using managed encryption.
AES-Encrypted Transports - Secures message payloads using AES-256-GCM encryption at both the producer and storage levels.
Data Pipeline Extensions - Allows creation of custom source, sink, or transformation logic by implementing specific traits for system integration.
Trait-Based Plugin Interfaces - Implements a modular plugin system using trait-based interfaces to create custom source and sink connectors.
Administrative Dashboards - Provides a web-based administrative dashboard to manage streams, topics, and user permissions.
CLI System Management - Offers a system management CLI featuring session-based login and named connection contexts.
Observability Dashboards - Aggregates system health metrics, OpenTelemetry traces, and Prometheus data into a unified observability dashboard.
System Health Monitoring - Retrieves server statistics, cluster metadata, and client connection details to troubleshoot system performance.
Web Dashboards - Ships a built-in web user interface to visualize server configuration, logs, and stream activity.

hazelcast/hazelcast

6,570View on GitHub

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

nats-io/nats-server

20,076View on GitHub

NATS Server is a high-performance, lightweight messaging system designed for cloud-native applications, edge computing, and distributed microservices. It functions as a distributed publish-subscribe broker that routes messages using hierarchical, dot-separated subject strings, enabling decoupled communication between services without requiring centralized broker lookups. The system supports core messaging patterns including asynchronous publish-subscribe, request-reply, and load-balanced queue processing. The platform distinguishes itself through a decentralized architecture that eliminates t

rabbitmq/rabbitmq-tutorials

6,889View on GitHub

This project is a collection of learning resources and instructional guides for implementing asynchronous messaging patterns using RabbitMQ. It provides a series of tutorials and runnable code examples focused on the Advanced Message Queuing Protocol to help users decouple services via a message broker. The resources cover practical implementation patterns including request-reply, pub-sub, and stream processing. These guides demonstrate how to use official client libraries to balance worker loads, route messages across multiple consumers in a distributed system, and deploy high availability b

infinyon/fluvio

5,231View on GitHub

Fluvio is a distributed event streaming platform and cloud-native streaming engine designed for collecting, persisting, and replicating real-time data streams across a distributed cluster. It functions as a real-time data pipeline for building stateful workflows that ingest, enrich, and export data between external sources and sinks. The platform is distinguished by its use of WebAssembly to execute compiled modules for in-line data transformations and filtering. This allows for the execution of custom business logic to reshape information in motion without requiring a restart of the cluster.

apacheiggy

Features