# High-Throughput Message Brokers and Streaming

> Search results for `message queue and event-streaming platform for high-throughput pipelines` on awesome-repositories.com. 119 total matches; showing the first 50.

Explore on the web: https://awesome-repositories.com/q/message-queue-and-event-streaming-platform-for-high-throughput-pipelines

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [this search on awesome-repositories.com](https://awesome-repositories.com/q/message-queue-and-event-streaming-platform-for-high-throughput-pipelines).**

## Results

- [apache/incubator-rocketmq](https://awesome-repositories.com/repository/apache-incubator-rocketmq.md) (22,461 ⭐) — RocketMQ is a distributed messaging and streaming platform designed for building event-driven applications. It serves as middleware to decouple services using publish-subscribe and request-reply patterns, and functions as a transactional messaging system that ensures atomicity by linking message delivery to local transaction outcomes.

The platform includes specialized capabilities as a Kubernetes-native message broker for container orchestration environments and an MQTT broker for ingesting event data from mobile applications and hardware terminals.

The system covers high-throughput data str
- [dominictarr/event-stream](https://awesome-repositories.com/repository/dominictarr-event-stream.md) (2,180 ⭐) — EventStream is like functional programming meets IO
- [dragonflydb/dragonfly](https://awesome-repositories.com/repository/dragonflydb-dragonfly.md) (30,688 ⭐) — Dragonfly is a high-performance, multi-model in-memory data store designed to serve as a drop-in replacement for existing database infrastructures. By utilizing a multi-threaded, shared-nothing architecture and a fiber-based concurrency model, it maximizes CPU utilization and minimizes latency for read and write operations. The system supports a wide range of data structures, including strings, hashes, lists, sets, sorted sets, and JSON documents, while maintaining full compatibility with standard industry wire protocols and client libraries.

What distinguishes Dragonfly is its focus on effic
- [anthropics/claude-code](https://awesome-repositories.com/repository/anthropics-claude-code.md) (132,728 ⭐) — Anthropic's terminal-native AI coding agent.
- [run-as-root/magento2-message-queue-retry](https://awesome-repositories.com/repository/run-as-root-magento2-message-queue-retry.md) (64 ⭐) — It gives the possibility to process the same queue message more than once, utilizing The RabbitMQ's dead letter exchange feature.
- [redpanda-data/redpanda](https://awesome-repositories.com/repository/redpanda-data-redpanda.md) (12,248 ⭐) — Redpanda is a distributed event streaming engine designed to serve as a high-performance, drop-in replacement for existing event-driven architectures. It provides a foundation for building and scaling applications that require reliable data movement, analytical querying, and strict operational compliance across both cloud and self-managed environments.

The platform distinguishes itself through a shared-nothing architecture that utilizes thread-per-core execution and a non-blocking asynchronous input/output engine to maximize throughput. It maintains data consistency through a consensus-based
- [apache/kafka](https://awesome-repositories.com/repository/apache-kafka.md) (32,846 ⭐) — Kafka is a distributed event streaming platform designed for capturing, storing, and processing real-time data streams across interconnected nodes. It functions as a distributed commit log, providing a fault-tolerant storage mechanism that records state changes sequentially to ensure data consistency and durability across distributed environments.

The platform distinguishes itself through a partitioned commit log architecture that enables horizontal scaling and parallel processing of data streams. It integrates a stream processing engine for continuous transformations and aggregations, while
- [langbot-app/langbot](https://awesome-repositories.com/repository/langbot-app-langbot.md) (15,311 ⭐) — LangBot is an orchestration platform designed for building, managing, and deploying AI agents. It functions as a comprehensive framework for integrating large language models with custom workflows, enabling developers to connect intelligent agents to various messaging platforms and external tools.

The platform distinguishes itself through a modular, plugin-based architecture that allows for the extension of agent capabilities via custom tools and file parsers. It features a secure, sandbox-isolated runtime environment that executes untrusted code and plugin logic within resource-constrained c
- [emersion/go-message](https://awesome-repositories.com/repository/emersion-go-message.md) (451 ⭐) — ✉️ A streaming Go library for the Internet Message Format and mail messages
- [bee-queue/bee-queue](https://awesome-repositories.com/repository/bee-queue-bee-queue.md) (4,032 ⭐) — A simple, fast, robust job/task queue for Node.js, backed by Redis.
- [redis/redis](https://awesome-repositories.com/repository/redis-redis.md) (74,906 ⭐) — Redis is an in-memory, key-value database designed to provide sub-millisecond latency for read and write operations. It functions as a versatile data platform, serving as a distributed cache, a message broker, a NoSQL document store, and a vector database. The system utilizes an event-driven, single-threaded loop to process requests efficiently, while maintaining data durability through append-only persistence logs and asynchronous snapshotting mechanisms.

What distinguishes Redis is its ability to handle complex data structures—including strings, hashes, lists, sets, and sorted sets—alongsid
- [activepieces/activepieces](https://awesome-repositories.com/repository/activepieces-activepieces.md) (20,887 ⭐) — Activepieces is an open-source, self-hosted workflow automation platform designed to connect third-party applications through modular triggers and actions. It provides a low-code integration framework that allows users to build, manage, and execute complex business logic sequences within isolated, sandboxed environments.

The platform distinguishes itself through its focus on embeddability and enterprise-grade security. It features an embedded automation builder that can be integrated into external applications via iframes, supported by comprehensive identity and access management tools such a
- [redis/go-redis](https://awesome-repositories.com/repository/redis-go-redis.md) (22,159 ⭐) — This project is a feature-rich Go client library designed for interacting with Redis. It serves as a comprehensive interface for managing remote data stores, enabling developers to execute standard database commands, handle complex data structures, and perform asynchronous operations within Go applications.

The library distinguishes itself through its support for advanced Redis capabilities, including connection pooling, pipelining, and transactional integrity. It provides specialized primitives for managing distributed clusters, including automated topology updates and request routing to sha
- [cakephp/queue](https://awesome-repositories.com/repository/cakephp-queue.md) (36 ⭐) — This is a Queue system for CakePHP.
- [conductor-oss/conductor](https://awesome-repositories.com/repository/conductor-oss-conductor.md) (31,962 ⭐) — Conductor is a durable workflow engine designed to orchestrate complex, long-running business processes and autonomous agent loops. It functions as a stateful execution platform that persists the entire history of a process, ensuring that workflows remain reliable and recoverable across infrastructure failures, system restarts, and transient network errors. By managing task lifecycles, worker polling, and state transitions, it provides a centralized coordination layer for distributed systems.

The platform distinguishes itself through its specialized support for AI agent orchestration, allowin
- [openhft/chronicle-queue](https://awesome-repositories.com/repository/openhft-chronicle-queue.md) (3,692 ⭐) — Chronicle Queue is a high-performance data handling system featuring off-heap message queues, memory-mapped file stores, and replicated message stores. It provides a binary compatible memory layout that enables different programming languages to share data without serialization overhead.

The system utilizes a replicated message store to synchronize data across multiple nodes, ensuring high availability and instant failover. Its memory-mapped architecture supports deterministic replay from disk and low-latency data recording.

The project implements off-heap memory management and zero-allocati
- [avelino/awesome-go](https://awesome-repositories.com/repository/avelino-awesome-go.md) (175,576 ⭐) — This project serves as a comprehensive language ecosystem index, functioning as a centralized, community-curated directory for the Go programming language. It organizes a vast landscape of software components, libraries, and development tools into a structured, navigable hierarchy, enabling developers to efficiently discover resources tailored to specific functional domains.

The repository distinguishes itself through a decentralized contribution model, where community-driven updates ensure the index remains current with the rapidly evolving software landscape. Beyond simple resource listing,
- [mountain-loop/yaak](https://awesome-repositories.com/repository/mountain-loop-yaak.md) (18,778 ⭐) — Yaak is a cross-platform desktop client and command-line utility designed for developing, testing, and debugging API endpoints. It supports multi-protocol request execution for REST, GraphQL, and gRPC services, providing a unified environment for managing network interactions, authentication credentials, and automated testing workflows.

The tool distinguishes itself through a local-first architecture that stores all workspace configurations and request definitions directly on the filesystem. This design enables native integration with version control systems like Git, allowing teams to track
- [nats-io/nats-server](https://awesome-repositories.com/repository/nats-io-nats-server.md) (20,076 ⭐) — NATS Server is a high-performance, lightweight messaging system designed for cloud-native applications, edge computing, and distributed microservices. It functions as a distributed publish-subscribe broker that routes messages using hierarchical, dot-separated subject strings, enabling decoupled communication between services without requiring centralized broker lookups. The system supports core messaging patterns including asynchronous publish-subscribe, request-reply, and load-balanced queue processing.

The platform distinguishes itself through a decentralized architecture that eliminates t
- [letta-ai/letta](https://awesome-repositories.com/repository/letta-ai-letta.md) (21,168 ⭐) — Letta is a framework for building, deploying, and managing autonomous AI agents that maintain persistent state across long-term interactions. It provides a comprehensive suite of primitives for defining agents with configurable personas, modular memory blocks, and tool-use capabilities, enabling them to retain user preferences and conversation history over extended sessions.

The platform distinguishes itself through its advanced memory management and orchestration capabilities. It allows agents to autonomously update their own memory, perform retrieval-augmented generation, and coordinate com
- [kubeflow/pipelines](https://awesome-repositories.com/repository/kubeflow-pipelines.md) (4,154 ⭐) — Machine Learning Pipelines for Kubeflow
- [tektoncd/pipeline](https://awesome-repositories.com/repository/tektoncd-pipeline.md) (8,996 ⭐) — The Tekton Pipelines project provides k8s-style resources for declaring CI/CD-style pipelines.
- [langchain-ai/langchainjs](https://awesome-repositories.com/repository/langchain-ai-langchainjs.md) (17,818 ⭐) — LangChain.js is a framework for building, executing, and monitoring stateful agentic applications. It provides an orchestration engine that models workflows as directed graphs, allowing developers to connect language models, data sources, and external tools into modular, multi-step processes.

The platform distinguishes itself through its focus on stateful execution and human-in-the-loop control. It manages agent lifecycles by persisting execution state across threads, enabling fault tolerance and the ability to pause workflows at designated breakpoints for manual review or modification. This
- [ibm/sarama](https://awesome-repositories.com/repository/ibm-sarama.md) (12,490 ⭐) — Sarama is an Apache Kafka Go client library that provides native support for the Kafka protocol. It includes a protocol client for managing offsets and timestamps, a producer implementation for sending messages, and a consumer group coordinator to balance workloads across multiple instances.

The library enables high throughput data streaming through concurrent message production and maintains strict partition ordering during network retries. It supports secure communication with Kafka brokers using certificate-based encryption to protect data traffic.

The project covers a broad range of dist
- [fosrl/pangolin](https://awesome-repositories.com/repository/fosrl-pangolin.md) (21,255 ⭐) — Pangolin is a zero-trust remote access platform designed to provide secure, identity-aware connectivity to private network resources. It functions as a cloud-native network controller that orchestrates encrypted tunnels, traffic routing, and access policies across distributed environments. By leveraging WireGuard for secure data transport, the platform enables authenticated access to internal web applications, terminal sessions, and remote desktops without exposing services to the public internet.

The platform distinguishes itself through a declarative infrastructure model that synchronizes n
- [shopify/sarama](https://awesome-repositories.com/repository/shopify-sarama.md) (12,490 ⭐) — Sarama is a Go client library for producing and consuming messages from Apache Kafka clusters. It provides dedicated interfaces for a message producer to send typed data packets and a message consumer to read and process continuous data streams from Kafka topics.

The library includes a mocking framework and simulation layer that mimics Kafka broker behavior. These tools allow for the testing of client applications and messaging logic without requiring a live cluster.

The project handles cluster connectivity and integration through the implementation of the Kafka binary protocol over TCP. It
- [functional-streams-for-scala/fs2](https://awesome-repositories.com/repository/functional-streams-for-scala-fs2.md) (2,447 ⭐) — Compositional, streaming I/O library for Scala
- [serengil/deepface](https://awesome-repositories.com/repository/serengil-deepface.md) (22,226 ⭐) — Deepface is a comprehensive deep learning library for facial recognition and demographic analysis. It provides a modular pipeline that handles the entire lifecycle of facial processing, including detection, geometric alignment, and the transformation of facial images into high-dimensional numerical vector embeddings for identity verification and similarity comparison.

The library distinguishes itself through a model ensemble approach, which combines predictions from multiple pre-trained neural networks to improve classification accuracy and reduce bias. It also integrates advanced security fe
- [kedacore/keda](https://awesome-repositories.com/repository/kedacore-keda.md) (10,314 ⭐) — KEDA is a Kubernetes event-driven autoscaler and cloud event scaling engine. It functions as a custom metrics provider that monitors external event sources—including message brokers, databases, and cloud metrics—to dynamically adjust the replica counts of containerized workloads.

The project is distinguished by its scale-to-zero workflow, which reduces workloads to zero replicas during inactivity and automatically restarts them when new events are detected. It operates as a multi-cloud event trigger system, using a pluggable scaler interface to integrate with a wide array of third-party servi
- [nestjstools/messaging](https://awesome-repositories.com/repository/nestjstools-messaging.md) (40 ⭐) — Simplifies asynchronous and synchronous message handling with support for buses, handlers, channels, and consumers. Build scalable, decoupled applications with ease and reliability.
- [segmentio/kafka-go](https://awesome-repositories.com/repository/segmentio-kafka-go.md) (8,581 ⭐) — Kafka library in Go
- [gradio-app/gradio](https://awesome-repositories.com/repository/gradio-app-gradio.md) (42,931 ⭐) — Gradio is a Python library that enables the creation of interactive web applications by converting functions into browser-based interfaces. It functions as a declarative framework where developers define input and output components to automatically generate web forms, visualizations, and data-driven dashboards. By abstracting away manual web markup, the library allows for the rapid construction of interfaces for machine learning models, research demonstrations, and analytical workflows within a single environment.

The platform distinguishes itself by automatically exposing internal applicatio
- [thephpleague/pipeline](https://awesome-repositories.com/repository/thephpleague-pipeline.md) (1,000 ⭐) — League\Pipeline
- [hyfather/pipeline](https://awesome-repositories.com/repository/hyfather-pipeline.md) (61 ⭐) — Pipelines using goroutines
- [rethinkdb/rethinkdb](https://awesome-repositories.com/repository/rethinkdb-rethinkdb.md) (26,996 ⭐) — RethinkDB is a distributed, document-oriented database designed to store and manage JSON-formatted data across scalable clusters. It utilizes a custom log-structured storage engine with B-Tree indexing to ensure high-performance disk I/O and data persistence. The system maintains high availability through automatic sharding and replication, employing a primary-replica voting consensus mechanism to handle node failures and ensure consistent cluster operations.

A defining characteristic of the platform is its reactive changefeed engine, which allows applications to subscribe to live data update
- [cockroachdb/cockroach](https://awesome-repositories.com/repository/cockroachdb-cockroach.md) (32,207 ⭐) — Cockroach is a distributed SQL database designed to scale horizontally across multiple nodes while maintaining strict ACID compliance and global data consistency. It functions as a relational database engine that automatically partitions data into ranges, rebalancing them across a cluster to accommodate growing storage and throughput requirements. By utilizing a distributed consensus protocol, the system ensures that all nodes agree on the order of operations, providing fault tolerance and continuous availability even in the event of hardware failures.

The system distinguishes itself through
- [quarkusio/quarkus](https://awesome-repositories.com/repository/quarkusio-quarkus.md) (15,479 ⭐) — Quarkus is a Kubernetes-native Java framework designed for building high-performance, memory-efficient applications. It utilizes ahead-of-time native compilation to transform Java code into standalone, optimized binaries that eliminate the need for a virtual machine, enabling rapid startup and reduced memory consumption. By performing code augmentation during the build phase, it shifts heavy processing tasks away from runtime, ensuring that applications are optimized for cloud-native environments.

The framework distinguishes itself through a unified approach to reactive and imperative program
- [adrianbrad/queue](https://awesome-repositories.com/repository/adrianbrad-queue.md) (357 ⭐) — ⏪️ Go package providing multiple queue implementations. Developed in a thread-safe generic way.
- [eczarny/spectacle](https://awesome-repositories.com/repository/eczarny-spectacle.md) (13,631 ⭐) — Spectacle is a keyboard-driven window manager and organizer that uses system accessibility frameworks to manipulate window coordinates and dimensions. It allows for the arrangement, resizing, and movement of application windows across multiple displays using global keyboard shortcuts.

The tool focuses on multi-monitor layout management, enabling users to shift active windows between connected displays and snap windows into predefined screen regions such as halves, thirds, or corners. It also provides the ability to center and maximize windows to optimize screen real estate without using a mou
- [emqx/emqx](https://awesome-repositories.com/repository/emqx-emqx.md) (16,422 ⭐) — This project is a high-performance MQTT broker and IoT data platform designed to manage millions of concurrent device connections. It provides a scalable infrastructure for ingesting, processing, and routing telemetry data across distributed systems, utilizing an actor-based concurrency model to maintain high availability and state synchronization across cluster nodes.

The platform distinguishes itself through integrated stream processing and edge computing capabilities. It allows users to execute declarative SQL-based rules directly against incoming message streams for real-time filtering, t
- [provectus/kafka-ui](https://awesome-repositories.com/repository/provectus-kafka-ui.md) (12,158 ⭐) — kafka-ui is a web interface and centralized control plane for administering Apache Kafka clusters, topics, and brokers. It functions as a distributed message queue dashboard and orchestrator, allowing for the oversight of multiple distributed Kafka environments from a single management interface.

The project provides dedicated tools for producing and inspecting messages within topics using various serialization and encoding formats. It includes a schema registry client for defining and versioning data schemas and a consumer monitoring dashboard to track offsets and calculate partition lag.

T
- [diamondio/better-queue](https://awesome-repositories.com/repository/diamondio-better-queue.md) (549 ⭐) — Better Queue for NodeJS
- [infisical/infisical](https://awesome-repositories.com/repository/infisical-infisical.md) (27,374 ⭐) — Infisical is a centralized secrets management platform designed to store, synchronize, and control access to sensitive credentials and configuration data across distributed development, staging, and production environments. It employs client-side encryption to ensure that secrets remain unreadable to the underlying storage infrastructure, while providing a hierarchical permission model to govern both user and machine access.

The platform distinguishes itself through dynamic credential provisioning, which generates short-lived access tokens that are automatically revoked after use. It supports
- [serilog/serilog](https://awesome-repositories.com/repository/serilog-serilog.md) (7,991 ⭐) — Serilog is a structured logging library for .NET applications that records events as rich data objects instead of plain text strings. It functions as a multi-sink framework, allowing a single log stream to be emitted to diverse destinations such as files, consoles, and cloud databases through a structured event pipeline.

The framework distinguishes itself through dynamic metadata enrichment, using log event enrichers to add contextual properties and telemetry to logs for improved traceability and analysis. It employs a routing system that filters, enriches, and directs these structured events
- [athanclark/purescript-queue](https://awesome-repositories.com/repository/athanclark-purescript-queue.md) (3 ⭐) — The pub/sub model captures a simple concept - assign "handlers" to an event manager of some kind, and delegate messages to those handlers by issuing events. This is also considered a "channel" in Haskell - something that stores messages until they are read.
- [getsentry/sentry](https://awesome-repositories.com/repository/getsentry-sentry.md) (44,108 ⭐) — This project is a comprehensive software observability suite and application performance monitoring platform designed to track runtime errors, performance bottlenecks, and system health. It functions as a centralized diagnostic service that aggregates and categorizes exceptions, providing the infrastructure necessary to visualize complex execution paths across distributed systems and microservices.

The platform distinguishes itself through a high-throughput distributed event ingestion pipeline and a columnar storage analytics engine that enables rapid aggregation of large-scale performance me
- [thephpleague/event](https://awesome-repositories.com/repository/thephpleague-event.md) (1,566 ⭐) — Event package for your app and domain
- [laravel/laravel](https://awesome-repositories.com/repository/laravel-laravel.md) (84,489 ⭐) — Laravel is a comprehensive full-stack web framework designed for building scalable server-side applications. It provides an integrated development environment that centers on an object-relational mapper for database abstraction, a robust routing system, and a sophisticated service container for dependency injection. The framework is built to handle complex application requirements through a modular architecture that emphasizes convention over configuration.

What distinguishes Laravel is its deep integration of background processing and event-driven communication. It features a task queue orch
- [laravel/framework](https://awesome-repositories.com/repository/laravel-framework.md) (34,774 ⭐) — This project is a full-stack web framework that provides a comprehensive environment for building server-side applications. It utilizes a model-view-controller architecture to separate application logic into distinct layers for data management, user interface presentation, and request handling. The platform manages the entire request-response lifecycle, including security, session handling, and background task processing, while using an object-relational mapping layer to translate database records into programmable objects.

The framework distinguishes itself through a central service containe
- [factual/durable-queue](https://awesome-repositories.com/repository/factual-durable-queue.md) (407 ⭐) — a disk-backed queue for clojure