Awesome GitHub RepositoriesProcess Sharding

xxl-job is a distributed task scheduling platform and job orchestrator designed to manage and trigger timed jobs across a cluster of remote executor nodes. It provides a centralized system for scheduling tasks, linking dependent jobs, and managing complex execution lifecycles through a relational database that persists configurations and logs. The platform distinguishes itself through a web-based interface for cron job management, allowing users to create and update scheduled tasks without modifying source code. It supports cross-language task execution by triggering logic on third-party exec

Discord.js is a Node.js library and framework for interacting with the Discord API. It provides a comprehensive set of wrappers for REST and WebSocket connections, enabling the development of automated server accounts and real-time chat applications. The project distinguishes itself through a distributed bot sharding system that splits a single bot instance across multiple processes to handle high server counts and large-scale workloads. It also includes a specialized voice API wrapper for managing audio streams and voice channel connectivity. The library covers broad capability areas includ

Redis is a high-performance in-memory key-value store that functions as a distributed cache, message broker, and NoSQL database. It provides sub-millisecond read and write access to data stored in RAM and can operate as a vector database for indexing high-dimensional embeddings. The system supports a wide range of data storage and synchronization primitives, including the management of strings, hashes, lists, sets, and JSON documents. It enables real-time data operations through atomic transactions, hybrid persistence using snapshots and append-only logs, and high-availability configurations

Cassandra is a distributed NoSQL database and wide-column store designed for high availability and linear scalability. It functions as a fault-tolerant distributed system that utilizes an LSM-tree storage engine to optimize write throughput and manage massive datasets. The system is a CQL-compliant database, using a structured query language to manage and retrieve tabular data stored across multiple nodes. It organizes information into rows and columns based on a flexible schema and primary keys. The project provides capabilities for horizontal database scaling, distributed data partitioning

This project provides educational materials and courseware focused on the theoretical and practical foundations of distributed systems design. It serves as a comprehensive curriculum covering the disciplines of consensus, data consistency, reliability engineering, and scalability. The instructional content focuses on achieving cluster agreement through consensus algorithms and managing system-wide state via coordination frameworks. It includes a dedicated guide to data theory, exploring replication strategies, consistency models, and data convergence. The courseware covers a broad capability

This project is a comprehensive knowledge base and study resource designed for mastering technical interviews. It provides structured guides, roadmaps, and curricula focused on data structures, algorithms, system design, and frontend engineering to help candidates prepare for software engineering screenings. The repository distinguishes itself by offering a holistic approach to professional advancement. Beyond technical drills, it includes a career development handbook covering resume optimization, salary benchmarking, and strategic negotiation coaching. It also provides detailed methodologie

This project is a comprehensive educational resource and curriculum focused on site reliability engineering, distributed systems, and infrastructure operations. It provides technical guides, a systems engineering course, and instructional manuals designed to teach the principles of managing large-scale computing environments. The curriculum covers high-level architectural design for scalability and resilience, including fault-tolerant infrastructure, high-availability patterns, and microservices decomposition. It emphasizes the practical application of site reliability engineering through the

kube-prometheus is a monitoring stack deployment and orchestration framework. It uses an operator pattern to automate the installation and lifecycle management of Prometheus and Alertmanager via custom resource definitions. The project focuses on scaling data collection through hash-based target sharding and topology-aware distribution to reduce cross-zone traffic. It implements a sidecar-based configuration reloading mechanism and utilizes consistent hashing to distribute scrape targets across multiple instances. The system covers broad observability capabilities including metric data colle

oneAPI Threading Building Blocks (oneTBB)

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

Pinot is a distributed, columnar analytical database designed for high-concurrency, low-latency query processing. It functions as a real-time OLAP datastore, enabling interactive, user-facing analytics by ingesting and querying massive datasets from both streaming and batch sources. The system architecture relies on a centralized controller for cluster coordination and a distributed segment-based storage model to ensure horizontal scalability. The platform distinguishes itself through a hybrid ingestion pipeline that unifies real-time event streams and historical batch data into a single quer

GreptimeDB is a distributed, open-source time-series database built for unified observability. It stores and queries metrics, logs, and traces together in a single columnar engine, supporting both SQL and PromQL for analysis. The database is designed as a Kubernetes-native operator with a decoupled compute and storage architecture, enabling horizontal scaling and multi-region deployment. What distinguishes GreptimeDB is its role as a multi-protocol ingestion gateway, accepting data through OpenTelemetry, Prometheus Remote Write, InfluxDB, Loki, Elasticsearch, Kafka, and MQTT protocols without

Discordgo is a Go library for building Discord bots and integrations, providing a complete client for both the Discord REST API and the real-time Gateway WebSocket protocol. It serves as a foundational package for developers who need to connect authenticated clients to Discord servers, send messages, handle events, and manage real-time communication programmatically. The library distinguishes itself through a comprehensive set of architectural components designed for production bot development. It includes a command routing middleware system that parses incoming messages and dispatches them t

Fluvio ist eine verteilte Event-Streaming-Plattform und eine Cloud-native Streaming-Engine, die für das Sammeln, Persistieren und Replizieren von Echtzeit-Datenströmen über einen verteilten Cluster hinweg entwickelt wurde. Sie fungiert als Echtzeit-Datenpipeline für den Aufbau zustandsbehafteter Workflows, die Daten zwischen externen Quellen und Senken aufnehmen, anreichern und exportieren. Die Plattform zeichnet sich durch die Verwendung von WebAssembly zur Ausführung kompilierter Module für In-Line-Datentransformationen und -filterung aus. Dies ermöglicht die Ausführung benutzerdefinierter Geschäftslogik, um Informationen während der Übertragung umzuformen, ohne den Cluster neu starten zu müssen. Das System deckt ein breites Spektrum an Funktionen ab, einschließlich connector-basierter Datenaufnahme aus externen Protokollen, log-strukturierter unveränderlicher Speicherung mit Zero-Copy-IO und horizontaler Clusterskalierung. Es unterstützt die Erstellung komplexer ereignisgesteuerter Pipelines, die zustandsbehaftete Verarbeitung, fensterbasierte Aggregationen und partitionierte Datenverteilung nutzen. Die Engine kann als leichtgewichtiges Binärprogramm auf diversen Systemarchitekturen bereitgestellt werden, einschließlich ARM64-IoT-Geräten für die Datenverarbeitung am Edge.

Titan ist eine verteilte Graphdatenbank und Computing-Engine, die für das Speichern und Abfragen massiver Datensätze aus miteinander verbundenen Knoten und Kanten über Multi-Maschinen-Cluster hinweg entwickelt wurde. Sie fungiert als skalierbare Graph-Speicherschicht und transaktionaler Speicher und bietet ein Framework für die Ausführung großskaliger Graph-Verarbeitungsjobs und tiefer Traversierungen. Das System zeichnet sich durch sein austauschbares Speicher-Backend aus, das die Graph-Engine von der physischen Persistenzschicht entkoppelt. Es nutzt Vertex-Cut-Datenpartitionierung, um Verarbeitungslasten auszugleichen, sowie ein Set-Kardinalitäts-Eigenschaftsmodell, das es ermöglicht, dass einzelne Eigenschaften mehrere Werte speichern können. Die Plattform deckt ein breites Spektrum an Funktionen ab, einschließlich Multi-Modell-Graph-Indizierung für geografische und Volltextsuchen, globales Schema-Management für die Neuindizierung von Datensätzen und transaktionale Operationen, die durch Write-Ahead-Logging sichergestellt werden. Zudem integriert es Element-Ablauf mittels Time-to-Live-Einstellungen und System-Performance-Monitoring zur Verfolgung von Abfrageaktivitäten und Transaktionslatenz.

WeNet is an end-to-end automatic speech recognition (ASR) toolkit designed for both Chinese and English, built around transformer-based models. It supports streaming and non-streaming inference out of the box, and is structured to be production-ready, with model export and deployment paths for servers and mobile devices. The toolkit distinguishes itself through a chunk-based streaming transformer architecture that processes audio in fixed-size segments for low latency while preserving context across chunks. It jointly trains models with both CTC and attention loss to combine alignment accurac

JDA is a Java wrapper for the Discord API, providing a framework for building automated agents and integrating applications with the chat platform. It functions as both a REST API client for managing server entities and a gateway client that maintains persistent WebSocket connections for real-time state updates. The library features a comprehensive voice API module for managing audio streaming, voice channel connectivity, and soundboard integration. It also includes a specialized toolkit for interactive UI design, enabling the creation of slash commands, buttons, select menus, and modal input

JDA is a Java library that serves as a Discord API wrapper, bot framework, and voice API client. It enables applications to interact with the Discord REST API and real-time gateway to build automated assistants, manage servers, and handle user interactions. The project includes a sharding manager to distribute gateway connections across multiple shards for scaling high-volume deployments. It also provides a UI component library for constructing interactive buttons, select menus, and modal forms within messages. Its capabilities cover voice and audio management, including audio streaming and

Zeebe ist eine Cloud-native Workflow-Engine und eine verteilte Zustandsmaschine, die für die Orchestrierung von Geschäftsprozessen unter Verwendung von BPMN- und DMN-Standards konzipiert wurde. Sie arbeitet als hochperformante gRPC-Workflow-Runtime, die komplexe Geschäftsprozesse durch eine partitionierte Event-Streaming-Architektur ausführt. Das System fungiert zudem als Orchestrator für Large-Language-Model-Agenten und koordiniert KI-Reasoning und Tool-Nutzung innerhalb deterministischer Geschäftsprozesse. Die Engine zeichnet sich durch ihr Peer-to-Peer-Broker-Networking und ein konsensbasiertes Datenreplikationsmodell aus, das hohe Verfügbarkeit und Fehlertoleranz sicherstellt. Sie setzt einen partitionierten Broker-Cluster ein, um horizontale Skalierbarkeit zu erreichen, und nutzt adaptives Request-Backpressure, um den eingehenden Befehlsfluss zu regulieren und Systemüberlastungen zu verhindern. Die Plattform deckt ein breites Spektrum operativer Funktionen ab, einschließlich Echtzeit-Ausführungsüberwachung mit Performance-Heatmaps, automatisierter Geschäftsentscheidungsfindung über Entscheidungstabellen und verteilter Task-Ausführung durch ein polling-basiertes Job-Worker-Modell. Sie bietet zudem Tools für Multi-Tenant-Ressourcenisolierung, identitätsbasierte Zugriffskontrolle und die Integration externer Web-APIs und serverloser Funktionen. Das System kann über verschiedene Umgebungen hinweg bereitgestellt werden, einschließlich Kubernetes und Docker, und wird über eine Kombination aus Kommandozeilenschnittstelle und programmatischer REST-API verwaltet.

RavenDB is a multi-model NoSQL document database designed for high-performance, ACID-compliant data storage. It persists structured information as schema-flexible JSON documents and utilizes a unit-of-work session pattern to track entity changes and batch modifications into atomic transactions. The platform is built on a distributed architecture that supports horizontal scaling through sharding and ensures high availability via multi-node, master-to-master cluster replication. The database distinguishes itself through a self-optimizing query engine that automatically creates and maintains ind