Awesome GitHub RepositoriesTask Queue Management

ToolJet is a low-code development platform designed for building and deploying internal business applications. It provides a visual interface where users can drag and drop components to design layouts, connect to various data sources, and execute custom logic. The platform is built on a containerized architecture, ensuring that applications remain portable and consistent across different cloud and server environments. The platform distinguishes itself through integrated artificial intelligence capabilities that assist in the generation of user interfaces, database schemas, and data queries fr

Deepagents is an LLM agent orchestration platform and stateful application server designed for deploying and managing AI agents built with computational graphs. It provides a containerized runtime environment that handles agent execution, state persistence, and the versioning of AI assistants. The platform distinguishes itself through deep integration with the Model Context Protocol, allowing agents to function as servers that expose tools and capabilities to external clients. It features a sophisticated observability suite for capturing execution traces, performing LLM-based evaluations agai

Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob

Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep

DouyinLiveRecorder is a multi-platform live stream recorder designed to monitor and save broadcasts from various global streaming platforms to local storage. It functions as an automated stream watchdog that continuously tracks broadcast statuses to initiate recording immediately when a stream goes live. The tool is a proxy-aware capture system capable of routing network requests through external servers to access and record content restricted to specific geographic regions. It utilizes platform-specific resolvers to map API patterns to stream identifiers, ensuring content can be archived fro

RoadRunner is a high-performance application server and process manager designed to serve PHP applications using a persistent worker model. It eliminates bootload overhead and initialization time by keeping application processes alive between requests, acting as a protocol-agnostic proxy that routes traffic to a pool of supervised workers. The server is built with a plugin-based modular architecture, allowing it to be extended with custom Go plugins and compiled into tailored binaries. It distinguishes itself by providing a unified execution model for a wide array of communication protocols,

Flower is a monitoring and administration tool for Celery task queues. It provides a real-time web dashboard and a REST API to monitor distributed task clusters, manage worker instances, and observe message broker health. The project distinguishes itself by offering centralized control over the task lifecycle, allowing users to trigger, revoke, or terminate tasks and apply execution rate limits. It also includes a Prometheus metrics exporter to surface internal performance and status data for external monitoring and alerting systems. The tool covers a broad range of observability and managem

PraisonAI is an autonomous AI agent platform that coordinates multiple LLM-powered agents for research, planning, and execution of complex workflows. It functions as a multi-agent orchestration framework, a workflow builder, and a Model Context Protocol server, while also providing retrieval-augmented generation through vector knowledge bases. Agents can interact via CLI, web, or standardized protocols with sandboxed code execution. The platform distinguishes itself with a rich set of agent communication protocols, including A2A, REST, WebSocket, voice and telephony integration, and MCP, allo

Agenta is a Prompt Ops lifecycle manager and prompt management platform that decouples prompt engineering from application code. It serves as a centralized system for developing, versioning, and deploying prompt templates and model configurations across different environments. The platform functions as an AI agent orchestrator with a visual interface for building agent workflows and connecting models to external tools. It further acts as an evaluation framework and observability tool, utilizing OpenTelemetry to capture execution traces, monitor latency, and track token costs. The system cove

Good Job ist ein Hintergrund-Job-Prozessor für Ruby on Rails, der eine PostgreSQL-Datenbank als primäre Speicher-Engine nutzt. Durch die Nutzung relationaler Datenbanktransaktionen stellt er eine persistente und zuverlässige Aufgabenausführung sicher und integriert sich direkt in das Active-Job-Framework, um asynchrone Operationen und wiederkehrende Job-Zeitpläne innerhalb bestehender Anwendungsumgebungen zu handhaben. Das System zeichnet sich durch ein In-Process-Ausführungsmodell aus, das es Hintergrund-Workern ermöglicht, im selben Prozess wie der Webserver zu laufen, was die Bereitstellung durch den Wegfall separater Worker-Dienste vereinfacht. Es nutzt multithreaded Worker-Ausführung und datenbankbasierte Advisory-Locks, um Aufgaben über verteilte Prozesse hinweg zu koordinieren, was eine eindeutige Ausführung für wiederkehrende Jobs und eine effiziente Ressourcennutzung sicherstellt. Die Bibliothek bietet umfassende betriebliche Kontrollen, einschließlich der Möglichkeit, verwandte Aufgaben in Batches für die kollektive Lebenszyklusverfolgung zu gruppieren und Masseneinfügungen zur Optimierung hochfrequenter Aufgabenaufnahmen zu nutzen. Administratoren können Nebenläufigkeitslimits verwalten, dedizierte Thread-Pools für spezifische Warteschlangen zuweisen und den Systemzustand über ein integriertes, anpassbares webbasiertes Dashboard überwachen. Das Projekt enthält ein integriertes Interface zum Inspizieren, Pausieren und Troubleshooting von Aufgaben in Echtzeit, neben konfigurierbarer Historienaufbewahrung für Auditing und Performance-Analysen.