Awesome GitHub RepositoriesTask Queue Management

ToolJet is a low-code development platform designed for building and deploying internal business applications. It provides a visual interface where users can drag and drop components to design layouts, connect to various data sources, and execute custom logic. The platform is built on a containerized architecture, ensuring that applications remain portable and consistent across different cloud and server environments. The platform distinguishes itself through integrated artificial intelligence capabilities that assist in the generation of user interfaces, database schemas, and data queries fr

Deepagents is an LLM agent orchestration platform and stateful application server designed for deploying and managing AI agents built with computational graphs. It provides a containerized runtime environment that handles agent execution, state persistence, and the versioning of AI assistants. The platform distinguishes itself through deep integration with the Model Context Protocol, allowing agents to function as servers that expose tools and capabilities to external clients. It features a sophisticated observability suite for capturing execution traces, performing LLM-based evaluations agai

Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob

Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep

DouyinLiveRecorder is a multi-platform live stream recorder designed to monitor and save broadcasts from various global streaming platforms to local storage. It functions as an automated stream watchdog that continuously tracks broadcast statuses to initiate recording immediately when a stream goes live. The tool is a proxy-aware capture system capable of routing network requests through external servers to access and record content restricted to specific geographic regions. It utilizes platform-specific resolvers to map API patterns to stream identifiers, ensuring content can be archived fro

RoadRunner is a high-performance application server and process manager designed to serve PHP applications using a persistent worker model. It eliminates bootload overhead and initialization time by keeping application processes alive between requests, acting as a protocol-agnostic proxy that routes traffic to a pool of supervised workers. The server is built with a plugin-based modular architecture, allowing it to be extended with custom Go plugins and compiled into tailored binaries. It distinguishes itself by providing a unified execution model for a wide array of communication protocols,

Flower is a monitoring and administration tool for Celery task queues. It provides a real-time web dashboard and a REST API to monitor distributed task clusters, manage worker instances, and observe message broker health. The project distinguishes itself by offering centralized control over the task lifecycle, allowing users to trigger, revoke, or terminate tasks and apply execution rate limits. It also includes a Prometheus metrics exporter to surface internal performance and status data for external monitoring and alerting systems. The tool covers a broad range of observability and managem

PraisonAI is an autonomous AI agent platform that coordinates multiple LLM-powered agents for research, planning, and execution of complex workflows. It functions as a multi-agent orchestration framework, a workflow builder, and a Model Context Protocol server, while also providing retrieval-augmented generation through vector knowledge bases. Agents can interact via CLI, web, or standardized protocols with sandboxed code execution. The platform distinguishes itself with a rich set of agent communication protocols, including A2A, REST, WebSocket, voice and telephony integration, and MCP, allo

Agenta is a Prompt Ops lifecycle manager and prompt management platform that decouples prompt engineering from application code. It serves as a centralized system for developing, versioning, and deploying prompt templates and model configurations across different environments. The platform functions as an AI agent orchestrator with a visual interface for building agent workflows and connecting models to external tools. It further acts as an evaluation framework and observability tool, utilizing OpenTelemetry to capture execution traces, monitor latency, and track token costs. The system cove

Good Job este un procesor de joburi în fundal pentru Ruby on Rails care utilizează o bază de date PostgreSQL ca motor principal de stocare. Prin valorificarea tranzacțiilor bazelor de date relaționale, asigură execuția persistentă și fiabilă a sarcinilor, integrându-se direct cu framework-ul Active Job pentru a gestiona operațiunile asincrone și programarea joburilor recurente în mediile de aplicație existente. Sistemul se distinge printr-un model de execuție in-process care permite worker-ilor din fundal să ruleze în același proces ca serverul web, simplificând deployment-ul prin eliminarea nevoii de servicii worker separate. Utilizează execuția worker-ilor multithreaded și lock-uri consultative la nivel de bază de date pentru a coordona sarcinile între procesele distribuite, asigurând execuția unică pentru joburile recurente și utilizarea eficientă a resurselor. Biblioteca oferă controale operaționale cuprinzătoare, inclusiv capacitatea de a grupa sarcini conexe în loturi pentru urmărirea colectivă a ciclului de viață și utilizarea inserției în masă pentru a optimiza ingestia sarcinilor de înaltă frecvență. Administratorii pot gestiona limitele de concurență, pot aloca pool-uri de thread-uri dedicate pentru cozi specifice și pot monitoriza starea sistemului printr-un dashboard web personalizabil și integrat. Proiectul include o interfață încorporată pentru inspectarea, pauzarea și depanarea sarcinilor în timp real, alături de retenția configurabilă a istoricului pentru audit și analiza performanței.