Awesome GitHub RepositoriesProcess Sharding

xxl-job is a distributed task scheduling platform and job orchestrator designed to manage and trigger timed jobs across a cluster of remote executor nodes. It provides a centralized system for scheduling tasks, linking dependent jobs, and managing complex execution lifecycles through a relational database that persists configurations and logs. The platform distinguishes itself through a web-based interface for cron job management, allowing users to create and update scheduled tasks without modifying source code. It supports cross-language task execution by triggering logic on third-party exec

Discord.js is a Node.js library and framework for interacting with the Discord API. It provides a comprehensive set of wrappers for REST and WebSocket connections, enabling the development of automated server accounts and real-time chat applications. The project distinguishes itself through a distributed bot sharding system that splits a single bot instance across multiple processes to handle high server counts and large-scale workloads. It also includes a specialized voice API wrapper for managing audio streams and voice channel connectivity. The library covers broad capability areas includ

Redis is a high-performance in-memory key-value store that functions as a distributed cache, message broker, and NoSQL database. It provides sub-millisecond read and write access to data stored in RAM and can operate as a vector database for indexing high-dimensional embeddings. The system supports a wide range of data storage and synchronization primitives, including the management of strings, hashes, lists, sets, and JSON documents. It enables real-time data operations through atomic transactions, hybrid persistence using snapshots and append-only logs, and high-availability configurations

Cassandra is a distributed NoSQL database and wide-column store designed for high availability and linear scalability. It functions as a fault-tolerant distributed system that utilizes an LSM-tree storage engine to optimize write throughput and manage massive datasets. The system is a CQL-compliant database, using a structured query language to manage and retrieve tabular data stored across multiple nodes. It organizes information into rows and columns based on a flexible schema and primary keys. The project provides capabilities for horizontal database scaling, distributed data partitioning

This project provides educational materials and courseware focused on the theoretical and practical foundations of distributed systems design. It serves as a comprehensive curriculum covering the disciplines of consensus, data consistency, reliability engineering, and scalability. The instructional content focuses on achieving cluster agreement through consensus algorithms and managing system-wide state via coordination frameworks. It includes a dedicated guide to data theory, exploring replication strategies, consistency models, and data convergence. The courseware covers a broad capability

This project is a comprehensive knowledge base and study resource designed for mastering technical interviews. It provides structured guides, roadmaps, and curricula focused on data structures, algorithms, system design, and frontend engineering to help candidates prepare for software engineering screenings. The repository distinguishes itself by offering a holistic approach to professional advancement. Beyond technical drills, it includes a career development handbook covering resume optimization, salary benchmarking, and strategic negotiation coaching. It also provides detailed methodologie

This project is a comprehensive educational resource and curriculum focused on site reliability engineering, distributed systems, and infrastructure operations. It provides technical guides, a systems engineering course, and instructional manuals designed to teach the principles of managing large-scale computing environments. The curriculum covers high-level architectural design for scalability and resilience, including fault-tolerant infrastructure, high-availability patterns, and microservices decomposition. It emphasizes the practical application of site reliability engineering through the

kube-prometheus is a monitoring stack deployment and orchestration framework. It uses an operator pattern to automate the installation and lifecycle management of Prometheus and Alertmanager via custom resource definitions. The project focuses on scaling data collection through hash-based target sharding and topology-aware distribution to reduce cross-zone traffic. It implements a sidecar-based configuration reloading mechanism and utilizes consistent hashing to distribute scrape targets across multiple instances. The system covers broad observability capabilities including metric data colle

oneAPI Threading Building Blocks (oneTBB)

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

Pinot is a distributed, columnar analytical database designed for high-concurrency, low-latency query processing. It functions as a real-time OLAP datastore, enabling interactive, user-facing analytics by ingesting and querying massive datasets from both streaming and batch sources. The system architecture relies on a centralized controller for cluster coordination and a distributed segment-based storage model to ensure horizontal scalability. The platform distinguishes itself through a hybrid ingestion pipeline that unifies real-time event streams and historical batch data into a single quer

GreptimeDB is a distributed, open-source time-series database built for unified observability. It stores and queries metrics, logs, and traces together in a single columnar engine, supporting both SQL and PromQL for analysis. The database is designed as a Kubernetes-native operator with a decoupled compute and storage architecture, enabling horizontal scaling and multi-region deployment. What distinguishes GreptimeDB is its role as a multi-protocol ingestion gateway, accepting data through OpenTelemetry, Prometheus Remote Write, InfluxDB, Loki, Elasticsearch, Kafka, and MQTT protocols without

Discordgo is a Go library for building Discord bots and integrations, providing a complete client for both the Discord REST API and the real-time Gateway WebSocket protocol. It serves as a foundational package for developers who need to connect authenticated clients to Discord servers, send messages, handle events, and manage real-time communication programmatically. The library distinguishes itself through a comprehensive set of architectural components designed for production bot development. It includes a command routing middleware system that parses incoming messages and dispatches them t

Fluvio is a distributed event streaming platform and cloud-native streaming engine designed for collecting, persisting, and replicating real-time data streams across a distributed cluster. It functions as a real-time data pipeline for building stateful workflows that ingest, enrich, and export data between external sources and sinks. The platform is distinguished by its use of WebAssembly to execute compiled modules for in-line data transformations and filtering. This allows for the execution of custom business logic to reshape information in motion without requiring a restart of the cluster.

Titan is a distributed graph database and computing engine designed for storing and querying massive datasets of interconnected nodes and edges across multi-machine clusters. It functions as a scalable graph storage layer and transactional store, providing a framework for executing large-scale graph processing jobs and deep traversals. The system is distinguished by its pluggable storage backend, which decouples the graph engine from the physical persistence layer. It utilizes vertex-cut data partitioning to balance processing loads and a set-cardinality property model that allows single prop

WeNet is an end-to-end automatic speech recognition (ASR) toolkit designed for both Chinese and English, built around transformer-based models. It supports streaming and non-streaming inference out of the box, and is structured to be production-ready, with model export and deployment paths for servers and mobile devices. The toolkit distinguishes itself through a chunk-based streaming transformer architecture that processes audio in fixed-size segments for low latency while preserving context across chunks. It jointly trains models with both CTC and attention loss to combine alignment accurac

JDA 是一个 Discord API 的 Java 包装器，提供了一个用于构建自动化智能体并将应用程序与聊天平台集成的框架。它既作为用于管理服务器实体的 REST API 客户端，也作为维护持久化 WebSocket 连接以获取实时状态更新的网关客户端。 该库具有全面的语音 API 模块，用于管理音频流、语音频道连接和音板集成。它还包括一个用于交互式 UI 设计的专用工具包，支持创建斜杠命令、按钮、选择菜单和模态输入表单。 广泛地，该项目涵盖了通过自动化规则管理进行的社区审核、通过频道和角色配置进行的服务器管理，以及使用分片（sharding）分配事件负载的流量管理。它还提供了用于富媒体内容创建和详细成员资料检索的实用工具。

JDA 是一个 Java 库，充当 Discord API 包装器、机器人框架和语音 API 客户端。它使应用程序能够与 Discord REST API 和实时网关交互，以构建自动化助手、管理服务器并处理用户交互。 该项目包括一个分片管理器，用于将网关连接分布在多个分片上，以扩展高容量部署。它还提供了一个 UI 组件库，用于在消息中构建交互式按钮、选择菜单和模态表单。 其能力涵盖语音和音频管理，包括音频流和音板播放。该库还包括用于社区管理、服务器管理以及通过基于规则的触发器进行自动化内容审核的工具。 该系统实现了多种架构模式，包括事件驱动的监听器注册表、可配置的内存缓存，以及针对 API 请求的自动速率限制处理。

Zeebe 是一个云原生工作流引擎和分布式状态机，旨在通过 BPMN 和 DMN 标准进行业务流程编排。它作为一个高性能 gRPC 工作流运行时，通过分区事件流架构执行复杂的业务流程。该系统还作为大语言模型代理的编排器，在确定性业务流程中协调 AI 推理和工具使用。 该引擎通过其点对点代理网络和确保高可用性和容错性的基于共识的数据复制模型而脱颖而出。它采用分区代理集群来实现水平扩展，并利用自适应请求背压来调节传入的命令流并防止系统过载。 该平台涵盖了广泛的操作功能，包括带有性能热力图的实时执行监控、通过决策表的自动化业务决策，以及通过基于轮询的作业工作者模型进行的分布式任务执行。它还提供用于多租户资源隔离、基于身份的访问控制以及集成外部 Web API 和无服务器函数的工具。 该系统可部署在 Kubernetes 和 Docker 等各种环境中，并通过命令行界面和程序化 REST API 的组合进行管理。

RavenDB is a multi-model NoSQL document database designed for high-performance, ACID-compliant data storage. It persists structured information as schema-flexible JSON documents and utilizes a unit-of-work session pattern to track entity changes and batch modifications into atomic transactions. The platform is built on a distributed architecture that supports horizontal scaling through sharding and ensures high availability via multi-node, master-to-master cluster replication. The database distinguishes itself through a self-optimizing query engine that automatically creates and maintains ind