apache/iggy

Features

• Append-Only Message Logs - Implements a high-throughput persistent append-only log for reading and writing raw binary data.
• Distributed Log Stores - Implements a persistent distributed log store using Viewstamped Replication for high availability and data consistency.
• LLM Streaming Infrastructure - Exposes a specialized server protocol that connects large language models to streaming data and operational system controls.
• Model Context Protocol Integrations - Provides standardized interfaces to link large language models to streaming data and system infrastructure.
• Data Ingestion Pipelines - Provides a modular system for ingesting and forwarding data between external sources and streams using plugins.
• Append-Only Storage Engines - Persists data as immutable sequences of records to enable multiple consumers to read and replay data.
• Distributed Data Replication - Synchronizes and replicates data across multiple cluster nodes using Viewstamped Replication for high availability.
• Consensus Replication Protocols - Ensures high availability and data consistency across a cluster using the Viewstamped Replication consensus protocol.
• Consumer Offset Trackers - Tracks the read progress of consumers using sequence positions (offsets) within data streams.
• Persistence & Durability - Ensures data persistence during crashes through filesystem synchronization and configurable background flush intervals.
• Horizontal Scaling - Distributes data across multiple partitions to increase parallel processing and enable horizontal scaling.
• Consumer Group Scaling - Enables horizontal scaling of message consumption through coordinated consumer groups and partition rebalancing.
• Partition Distribution Strategies - Distributes data across partitions using round-robin or key-based hashing to optimize message polling and sending.
• Message Consumption - Provides mechanisms for producing and consuming messages with support for partitioning and offset management.
• Binary Protocols - Uses a custom binary protocol for efficient data transmission across TCP, QUIC, WebSocket, and HTTP transports.
• Consumer Group Coordinators - Coordinates multiple consumers in groups to ensure partition-level exclusivity and horizontal scaling.
• Message Brokers - Functions as a multi-protocol message broker supporting data transmission via TCP, QUIC, HTTP, and WebSocket.
• Stream Offset Tracking - Manages message delivery by tracking consumer and server offsets within a stream.
• Stream Topic Publishing - Sends records to specific streams and partitions using optimized transport protocols for high-throughput data ingestion.
• Multi-Format Message Serialization - Supports multiple serialization formats including JSON, Protocol Buffers, and FlatBuffers for efficient binary storage.
• Multi-Protocol Transport Layers - Routes binary messages via a multi-protocol transport layer supporting TCP, QUIC, HTTP, and WebSockets.
• Multi-Transport Support - Supports multiple communication layers including TCP, QUIC, HTTP, and WebSockets to balance throughput and compatibility.
• Streaming Data Integrations - Encodes and decodes stream data using various binary and text formats.
• Permission-Based Access Control - Enforces a permission-based access control model to restrict administrative and data-polling actions.
• TLS Traffic Encryption - Secures network traffic across TCP, QUIC, HTTP, and WebSocket protocols using TLS encryption.
• User Account Management - Provides tools to create and manage user accounts and assign platform permissions.
• Control and Data Plane Separation - Separates replication into a control plane for managing users and topics and a data plane for streaming messages.
• Thread-Per-Core Architectures - Uses a thread-per-core architecture to map shards to specific CPU cores, improving hardware utilization and throughput.
• Replicated State Machines - Maintains identical state across cluster nodes by applying an ordered sequence of transactions via a consensus protocol.
• LLM Application Infrastructure - Provides backend infrastructure to connect large language models to streaming data through standard network transports.
• Stream Operation Controllers - Exposes a specialized server protocol that enables large language models to control message streaming operations.
• Stream Administration - Provides administrative client methods to create and configure streams, topics, and partitions.
• Data Compression Algorithms - Implements multiple compression algorithms including gzip, lz4, and zstd to optimize on-disk storage capacity.
• Partition Index Caching - Manages how partition indexes are stored in memory or read from disk to balance memory usage and access speed.
• Topic Management - Allows the creation and organization of named channels to distribute and partition streaming data.
• Custom Connector Development - Enables the development of custom input and output adapters to integrate the platform with external data sources.
• Data Pipeline Orchestration - Orchestrates data movement between external sources and systems using modular plugins.
• Data Retention Policies - Provides automated data deletion based on configurable time or size policies to manage disk usage.
• External Data Ingestion - Polls data from external systems and pushes it into streams via modular source plugins.
• Stream Data Exporters - Forwards messages from streams to external destinations using offset-tracking plugins.
• External System Integrations - Bridges streams with external databases and search engines using a plugin system with built-in transformations.
• Data Stream Management - Manages the full lifecycle of append-only streams, including updates, deletions, and purging of partitions.
• Topic Partition Mapping - Controls how logical topics are partitioned and mapped to distribute message loads across the cluster.
• CLI Administration Tools - Provides a command-line interface for administering streams, topics, partitions, and message flow.
• Control and Data Plane Separation - Architecturally separates replication into a control plane for metadata and a data plane for high-speed streaming.
• Cross-Shard Connection Migration - Moves client connections to the shard owning the target partition to remove the overhead of cross-shard messaging.
• AI Messaging Bridges - Provides integration layers that bridge message streaming infrastructure to large language models via HTTP or standard IO.
• Stream Payload Transformations - Modifies message fields by adding, deleting, or updating values before data enters or leaves a stream.
• CPU Affinity Binding - Binds shards to specific CPU cores and detects hardware topology to maximize processing efficiency.
• AES-GCM Data Encryptors - Provides AES-GCM encryption to secure sensitive message data at rest.
• Transit and At-Rest Encryption - Protects message payloads and state commands on disk using managed encryption.
• AES-Encrypted Transports - Secures message payloads using AES-256-GCM encryption at both the producer and storage levels.
• Data Pipeline Extensions - Allows creation of custom source, sink, or transformation logic by implementing specific traits for system integration.
• Trait-Based Plugin Interfaces - Implements a modular plugin system using trait-based interfaces to create custom source and sink connectors.
• Administrative Dashboards - Provides a web-based administrative dashboard to manage streams, topics, and user permissions.
• CLI System Management - Offers a system management CLI featuring session-based login and named connection contexts.
• Observability Dashboards - Aggregates system health metrics, OpenTelemetry traces, and Prometheus data into a unified observability dashboard.
• System Health Monitoring - Retrieves server statistics, cluster metadata, and client connection details to troubleshoot system performance.
• Web Dashboards - Ships a built-in web user interface to visualize server configuration, logs, and stream activity.

Star history