Awesome GitHub RepositoriesProcess Sharding

xxl-job is a distributed task scheduling platform and job orchestrator designed to manage and trigger timed jobs across a cluster of remote executor nodes. It provides a centralized system for scheduling tasks, linking dependent jobs, and managing complex execution lifecycles through a relational database that persists configurations and logs. The platform distinguishes itself through a web-based interface for cron job management, allowing users to create and update scheduled tasks without modifying source code. It supports cross-language task execution by triggering logic on third-party exec

Discord.js is a Node.js library and framework for interacting with the Discord API. It provides a comprehensive set of wrappers for REST and WebSocket connections, enabling the development of automated server accounts and real-time chat applications. The project distinguishes itself through a distributed bot sharding system that splits a single bot instance across multiple processes to handle high server counts and large-scale workloads. It also includes a specialized voice API wrapper for managing audio streams and voice channel connectivity. The library covers broad capability areas includ

Redis is a high-performance in-memory key-value store that functions as a distributed cache, message broker, and NoSQL database. It provides sub-millisecond read and write access to data stored in RAM and can operate as a vector database for indexing high-dimensional embeddings. The system supports a wide range of data storage and synchronization primitives, including the management of strings, hashes, lists, sets, and JSON documents. It enables real-time data operations through atomic transactions, hybrid persistence using snapshots and append-only logs, and high-availability configurations

Cassandra is a distributed NoSQL database and wide-column store designed for high availability and linear scalability. It functions as a fault-tolerant distributed system that utilizes an LSM-tree storage engine to optimize write throughput and manage massive datasets. The system is a CQL-compliant database, using a structured query language to manage and retrieve tabular data stored across multiple nodes. It organizes information into rows and columns based on a flexible schema and primary keys. The project provides capabilities for horizontal database scaling, distributed data partitioning

This project provides educational materials and courseware focused on the theoretical and practical foundations of distributed systems design. It serves as a comprehensive curriculum covering the disciplines of consensus, data consistency, reliability engineering, and scalability. The instructional content focuses on achieving cluster agreement through consensus algorithms and managing system-wide state via coordination frameworks. It includes a dedicated guide to data theory, exploring replication strategies, consistency models, and data convergence. The courseware covers a broad capability

This project is a comprehensive knowledge base and study resource designed for mastering technical interviews. It provides structured guides, roadmaps, and curricula focused on data structures, algorithms, system design, and frontend engineering to help candidates prepare for software engineering screenings. The repository distinguishes itself by offering a holistic approach to professional advancement. Beyond technical drills, it includes a career development handbook covering resume optimization, salary benchmarking, and strategic negotiation coaching. It also provides detailed methodologie

This project is a comprehensive educational resource and curriculum focused on site reliability engineering, distributed systems, and infrastructure operations. It provides technical guides, a systems engineering course, and instructional manuals designed to teach the principles of managing large-scale computing environments. The curriculum covers high-level architectural design for scalability and resilience, including fault-tolerant infrastructure, high-availability patterns, and microservices decomposition. It emphasizes the practical application of site reliability engineering through the

kube-prometheus is a monitoring stack deployment and orchestration framework. It uses an operator pattern to automate the installation and lifecycle management of Prometheus and Alertmanager via custom resource definitions. The project focuses on scaling data collection through hash-based target sharding and topology-aware distribution to reduce cross-zone traffic. It implements a sidecar-based configuration reloading mechanism and utilizes consistent hashing to distribute scrape targets across multiple instances. The system covers broad observability capabilities including metric data colle

oneAPI Threading Building Blocks (oneTBB)

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

Pinot is a distributed, columnar analytical database designed for high-concurrency, low-latency query processing. It functions as a real-time OLAP datastore, enabling interactive, user-facing analytics by ingesting and querying massive datasets from both streaming and batch sources. The system architecture relies on a centralized controller for cluster coordination and a distributed segment-based storage model to ensure horizontal scalability. The platform distinguishes itself through a hybrid ingestion pipeline that unifies real-time event streams and historical batch data into a single quer

GreptimeDB is a distributed, open-source time-series database built for unified observability. It stores and queries metrics, logs, and traces together in a single columnar engine, supporting both SQL and PromQL for analysis. The database is designed as a Kubernetes-native operator with a decoupled compute and storage architecture, enabling horizontal scaling and multi-region deployment. What distinguishes GreptimeDB is its role as a multi-protocol ingestion gateway, accepting data through OpenTelemetry, Prometheus Remote Write, InfluxDB, Loki, Elasticsearch, Kafka, and MQTT protocols without

Discordgo is a Go library for building Discord bots and integrations, providing a complete client for both the Discord REST API and the real-time Gateway WebSocket protocol. It serves as a foundational package for developers who need to connect authenticated clients to Discord servers, send messages, handle events, and manage real-time communication programmatically. The library distinguishes itself through a comprehensive set of architectural components designed for production bot development. It includes a command routing middleware system that parses incoming messages and dispatches them t

Fluvio este o platformă distribuită de streaming de evenimente și un motor de streaming cloud-native conceput pentru colectarea, persistența și replicarea fluxurilor de date în timp real într-un cluster distribuit. Funcționează ca un pipeline de date în timp real pentru construirea fluxurilor de lucru cu stare care ingerează, îmbogățesc și exportă date între surse și destinații externe. Platforma se distinge prin utilizarea WebAssembly pentru a executa module compilate pentru transformări și filtrare de date în linie. Acest lucru permite execuția logicii de business personalizate pentru a remodela informațiile în mișcare fără a necesita o repornire a clusterului. Sistemul acoperă o gamă largă de capabilități, inclusiv ingestia de date bazată pe conectori din protocoale externe, stocare imutabilă structurată pe log-uri cu IO de tip zero-copy și scalarea orizontală a clusterului. Suportă crearea de pipeline-uri complexe bazate pe evenimente care utilizează procesarea cu stare, agregări pe ferestre și distribuția datelor bazată pe partiții. Motorul poate fi implementat ca un binar ușor pe diverse arhitecturi de sistem, inclusiv dispozitive IoT ARM64 pentru procesarea datelor la margine (edge).

Titan este o bază de date graf distribuită și un motor de calcul conceput pentru stocarea și interogarea seturilor masive de date de noduri și muchii interconectate în clustere multi-mașină. Funcționează ca un strat de stocare graf scalabil și un magazin tranzacțional, oferind un framework pentru executarea sarcinilor de procesare graf la scară largă și a traversărilor profunde. Sistemul se distinge prin backend-ul său de stocare pluggable, care decuplează motorul graf de stratul de persistență fizică. Utilizează partiționarea datelor de tip vertex-cut pentru a echilibra sarcinile de procesare și un model de proprietăți set-cardinality care permite proprietăților unice să stocheze mai multe valori. Platforma acoperă o gamă largă de capabilități, inclusiv indexarea graf multi-model pentru căutări geografice și full-text, gestionarea globală a schemei pentru re-indexarea seturilor de date și operațiuni tranzacționale asigurate prin write-ahead logging. De asemenea, încorporează expirarea elementelor prin setări de tip time-to-live și monitorizarea performanței sistemului pentru urmărirea activității de interogare și a latenței tranzacțiilor.

WeNet is an end-to-end automatic speech recognition (ASR) toolkit designed for both Chinese and English, built around transformer-based models. It supports streaming and non-streaming inference out of the box, and is structured to be production-ready, with model export and deployment paths for servers and mobile devices. The toolkit distinguishes itself through a chunk-based streaming transformer architecture that processes audio in fixed-size segments for low latency while preserving context across chunks. It jointly trains models with both CTC and attention loss to combine alignment accurac

JDA is a Java wrapper for the Discord API, providing a framework for building automated agents and integrating applications with the chat platform. It functions as both a REST API client for managing server entities and a gateway client that maintains persistent WebSocket connections for real-time state updates. The library features a comprehensive voice API module for managing audio streaming, voice channel connectivity, and soundboard integration. It also includes a specialized toolkit for interactive UI design, enabling the creation of slash commands, buttons, select menus, and modal input

JDA is a Java library that serves as a Discord API wrapper, bot framework, and voice API client. It enables applications to interact with the Discord REST API and real-time gateway to build automated assistants, manage servers, and handle user interactions. The project includes a sharding manager to distribute gateway connections across multiple shards for scaling high-volume deployments. It also provides a UI component library for constructing interactive buttons, select menus, and modal forms within messages. Its capabilities cover voice and audio management, including audio streaming and

Zeebe este un motor de workflow cloud-native și o mașină de stare distribuită concepută pentru orchestrarea proceselor de business folosind standardele BPMN și DMN. Operează ca un runtime de workflow gRPC de înaltă performanță care execută procese de business complexe printr-o arhitectură de event-streaming partiționată. Sistemul funcționează, de asemenea, ca un orchestrator pentru agenți de tip LLM, coordonând raționamentul AI și utilizarea instrumentelor în cadrul proceselor de business deterministe. Motorul se distinge prin rețeaua de brokeri peer-to-peer și un model de replicare a datelor bazat pe consens care asigură disponibilitate ridicată și toleranță la erori. Utilizează un cluster de brokeri partiționat pentru a obține scalabilitate orizontală și utilizează backpressure adaptiv pentru a regla fluxul de comenzi primite și a preveni supraîncărcarea sistemului. Platforma acoperă o suprafață largă de capabilități operaționale, inclusiv monitorizarea execuției în timp real cu hărți termice de performanță, luarea deciziilor de business automate prin tabele de decizie și execuția distribuită a sarcinilor printr-un model de job worker bazat pe polling. Oferă, de asemenea, instrumente pentru izolarea resurselor multi-tenant, controlul accesului bazat pe identitate și integrarea API-urilor web externe și a funcțiilor serverless. Sistemul poate fi implementat în diverse medii, inclusiv Kubernetes și Docker, și este gestionat printr-o combinație de interfață în linie de comandă și API REST programatic.

RavenDB is a multi-model NoSQL document database designed for high-performance, ACID-compliant data storage. It persists structured information as schema-flexible JSON documents and utilizes a unit-of-work session pattern to track entity changes and batch modifications into atomic transactions. The platform is built on a distributed architecture that supports horizontal scaling through sharding and ensures high availability via multi-node, master-to-master cluster replication. The database distinguishes itself through a self-optimizing query engine that automatically creates and maintains ind