Awesome GitHub RepositoriesHigh-Throughput Data Streaming

RocketMQ is a distributed messaging and streaming platform designed for building event-driven applications. It serves as middleware to decouple services using publish-subscribe and request-reply patterns, and functions as a transactional messaging system that ensures atomicity by linking message delivery to local transaction outcomes. The platform includes specialized capabilities as a Kubernetes-native message broker for container orchestration environments and an MQTT broker for ingesting event data from mobile applications and hardware terminals. The system covers high-throughput data str

AISystem is a comprehensive AI full-stack infrastructure project covering the entire pipeline from AI chip architecture to high-level training frameworks. It encompasses the development of AI compiler frameworks, inference engines, and distributed training orchestrators designed to coordinate workloads across a heterogeneous compute stack of CPUs, GPUs, and NPUs. The project focuses on the deep integration of software and hardware, employing software-hardware co-design to align tensor layouts with physical memory structures. It provides specialized capabilities for accelerating Transformer mo

Apache Pulsar is a cloud-native message queue and distributed publish-subscribe messaging system. It serves as a multi-tenant event streaming platform designed to route data streams for asynchronous communication between producers and consumers. The system distinguishes itself through geo-replication, synchronizing data across multiple geographic regions to ensure high availability and low latency. It implements a multi-tenant architecture that provides isolation and resource management for millions of independent topics. The platform covers high-throughput data streaming and event-driven da

RabbitMQ is a multi-protocol messaging broker that functions as an AMQP message broker, a clustered message queue, and a distributed message stream. It provides a server for translating and bridging communication between diverse messaging standards to connect heterogeneous systems. The system distinguishes itself through distributed broker clustering and federation, using shoveling mechanisms to synchronize data across geographically separate sites. It supports high-throughput, append-only logs for persisting and reading large sequences of messages for real-time processing. The broker covers

Sarama is an Apache Kafka Go client library that provides native support for the Kafka protocol. It includes a protocol client for managing offsets and timestamps, a producer implementation for sending messages, and a consumer group coordinator to balance workloads across multiple instances. The library enables high throughput data streaming through concurrent message production and maintains strict partition ordering during network retries. It supports secure communication with Kafka brokers using certificate-based encryption to protect data traffic. The project covers a broad range of dist

This project provides a lossless compression algorithm and a byte-oriented compression library designed for high-speed data reduction and maximum decompression speed. It functions as a stream-oriented compression engine, a software library for encoding and decoding data blocks, and a command-line tool for managing interoperable compressed frames. The system distinguishes itself through the use of predefined pattern dictionaries to improve compression ratios for small data sets and small packets. It supports multiple processing modes, including high-speed block compression for minimal latency

RisingWave is a cloud-native streaming database and real-time analytics engine that uses standard SQL to process continuous data streams. It functions as a streaming data lakehouse, combining the capabilities of a streaming SQL database with a platform that integrates streaming ingestion with open table formats. The system is distinguished by its use of the PostgreSQL wire protocol, allowing it to integrate with existing SQL tools and drivers. It employs a decoupled compute and storage architecture, persisting streaming state and materialized views in cloud object storage to enable independen

This project is a software-defined radio platform designed to capture, analyze, and broadcast radio frequency signals across a wide spectrum. It provides a programmable hardware interface for transmitting and receiving radio signals, enabling spectrum analysis and wireless data monitoring. The system is distinguished by its ability to synchronize multiple devices using a shared external clock and hardware triggers to ensure precise timing and sample accuracy. It supports advanced signal routing, allowing ports to be mapped based on frequency or time to enable specialized operations like pseud

This project is a collection of learning resources and instructional guides for implementing asynchronous messaging patterns using RabbitMQ. It provides a series of tutorials and runnable code examples focused on the Advanced Message Queuing Protocol to help users decouple services via a message broker. The resources cover practical implementation patterns including request-reply, pub-sub, and stream processing. These guides demonstrate how to use official client libraries to balance worker loads, route messages across multiple consumers in a distributed system, and deploy high availability b

Apache Storm is a distributed stream processing framework and real-time data processing engine. It functions as a fault-tolerant distributed computing system designed to analyze data in motion across a cluster of machines for continuous stream computation. The system enables the creation of fault-tolerant data pipelines and scalable event processing by distributing workloads across a network of computing nodes. This architecture ensures low latency and high throughput for live data while allowing the system to recover automatically from individual node failures. The framework provides capabi

DotNetty, .NET के लिए एक एसिंक्रोनस नेटवर्क फ्रेमवर्क और इवेंट-ड्रिवन नेटवर्किंग लाइब्रेरी है। यह Netty आर्किटेक्चर का एक पोर्ट है जिसे उच्च-प्रदर्शन वाले प्रोटोकॉल सर्वर और क्लाइंट बनाने के लिए डिज़ाइन किया गया है। यह फ्रेमवर्क ऐसे नेटवर्क एप्लिकेशनों के विकास को सक्षम बनाता है जो निष्पादन थ्रेड्स को ब्लॉक किए बिना समवर्ती कनेक्शन और उच्च-थ्रूपुट डेटा स्ट्रीमिंग को संभालते हैं। यह परिभाषित एन्कोडिंग और डिकोडिंग नियमों के माध्यम से कस्टम नेटवर्क प्रोटोकॉल के कार्यान्वयन का समर्थन करता है। यह लाइब्रेरी नेटवर्क ट्रैफ़िक को प्रबंधित करने के लिए पाइपलाइन-आधारित प्रोसेसिंग मॉडल और नॉन-ब्लॉकिंग I/O का उपयोग करती है। इसका आर्किटेक्चर एक इवेंट-लूप संचालित मॉडल, रिक्वेस्ट डिस्पैचिंग के लिए रिएक्टर पैटर्न, और मेमोरी को प्रबंधित करने के लिए रेफरेंस काउंटिंग के साथ एक कस्टम बफर सिस्टम को शामिल करता है।

franz-go is a low-level Go client library and wire protocol implementation for producing, consuming, and administering Kafka clusters. It functions as a zero-allocation network driver that utilizes a direct TCP communication layer to handle requests and responses. The project integrates a schema registry client for encoding and decoding structured data. It provides a programmatic interface for cluster administration, including the management of topics, access control lists, and broker configurations. The library covers data consumption through consumer groups, message production with transac