Awesome GitHub RepositoriesTask Distribution Scaling

Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages, following links, and extracting structured data from HTML and XML documents. The framework includes a distributed scraping engine designed to spread data collection tasks across multiple instances to increase throughput. It ensures compliance with website owner policies by automatically reading and respecting robots.txt files. The system manages request lifecycles through domain-based rate limiting, concurrency controls, and session management via a stateful cookie jar. It s

Hydro is an online judge platform and competitive programming management system. It provides the infrastructure to host programming contests, manage a library of programming problems, and evaluate code submissions against predefined test cases and time limits. The system utilizes a distributed code execution engine that scales judging tasks across multiple worker nodes to process high volumes of submissions. It is built as a modular judge framework, employing a plugin-based architecture that allows for the extension of system functionality without modifying the core source code. The platform

Hatchet is an open-source durable workflow engine and task orchestration platform. It provides a framework for building and executing fault-tolerant, multi-step pipelines as directed acyclic graphs (DAGs), with automatic retries, scheduling, and real-time observability. The system is built around durable task checkpointing, which persists execution state after each step so work can resume from the last checkpoint after a worker crash or restart, and it supports event-driven task resumption that pauses a task until a matching external event arrives. The platform distinguishes itself through it

Minigo is a TensorFlow-based reinforcement learning engine designed to master the game of Go. It functions as a comprehensive system for training neural networks to predict board policies and game outcomes, utilizing a model trainer to generate self-play data and optimize weights. The project is distinguished by its ability to perform large-scale game simulations using Kubernetes to distribute worker nodes across CPU, GPU, and TPU hardware. It employs a Monte Carlo Tree Search implementation to identify optimal moves and supports specialized hardware acceleration, including inference on Edge

This project is a distributed scraping engine designed to extract business details, customer reviews, and lead information from Google Maps. It functions as a business scraper and data extractor that can be deployed as a permanent system or as on-demand serverless functions. The system utilizes a proxy-routed web crawler to manage request origins via SOCKS5, HTTP, and HTTPS proxies. To locate contact information, it includes an email extraction tool that recursively crawls business websites linked within map listings. The software supports coordinate-based radius searches for efficient data

This platform serves as a centralized dashboard for collecting, visualizing, and analyzing personal activity logs and timestamped life events. It functions as a unified repository that aggregates disparate digital records into a single store, enabling long-term historical tracking and personal data analysis. The system distinguishes itself through a modular report composition engine that groups specific event datasets and visual elements into reusable structures. It incorporates an automated alerting engine that monitors incoming data streams against predefined thresholds, triggering notifica

This project provides a collection of Helm charts designed for deploying and managing self-hosted error tracking and performance monitoring infrastructure within a Kubernetes cluster. It enables the deployment of complex, multi-service application stacks using standardized configuration templates to ensure consistent and repeatable infrastructure management. The platform distinguishes itself by automating the operational requirements of distributed event processing, including the horizontal autoscaling of background task workers and the secure generation of cryptographic keys for internal ser