Awesome GitHub RepositoriesTask Queues

Paperless-ngx is a self-hosted document management server designed to transform physical paperwork into a searchable, organized digital archive. It functions as a private platform for storing, indexing, and retrieving documents, providing users with full control over their data on local infrastructure or private cloud servers. The system distinguishes itself through an automated workflow engine that categorizes, tags, and routes incoming files using content analysis and metadata extraction. To maintain responsiveness during resource-intensive tasks like optical character recognition, it utili

Seal is a mobile application designed to retrieve video and audio content from various online platforms. It functions as a graphical interface that manages background transfer processes, allowing users to download and archive media files directly to their local device storage for offline access. The application distinguishes itself by acting as a bridge to powerful command-line utilities, orchestrating these external binaries to handle complex media extraction and file conversion tasks. Users can customize their experience through a declarative template system that defines specific execution

ArchiveBox is a self-hosted archiving tool designed for personal digital preservation and research data management. It functions as an automated web preservation engine that monitors URL inputs from bookmarks, browser history, or manual entries to capture and store permanent, offline copies of web content. By utilizing headless browser automation, the system renders dynamic web pages to ensure that captured snapshots, PDFs, and media assets remain accurate and accessible even if the original source disappears. The project distinguishes itself through a modular extractor pipeline and a task-qu

Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob

This project provides a framework for managing multi-agent systems, designed to automate complex software development, infrastructure, and business workflows. It functions as a multi-agent workflow orchestrator that routes tasks to domain-specific workers while maintaining state persistence and infrastructure automation. By leveraging large language models, the system decomposes high-level objectives into actionable plans, ensuring that complex operations are executed with consistency and reliability. The framework distinguishes itself through its hierarchical agent registry and policy-driven

Temporal is a distributed workflow orchestration engine designed to manage fault-tolerant, stateful, and long-running background processes. It functions as a platform for coordinating complex cross-service operations, ensuring consistency and reliability in distributed environments by decoupling workflow orchestration from task execution. The platform distinguishes itself through a deterministic, event-sourced execution model that reconstructs workflow state by re-executing code from an immutable event log. This approach isolates non-deterministic side effects into managed activities, allowin

DataLoader is a utility that collects individual data loads into a single batch and caches results to minimize redundant backend requests. It operates on a batch-and-cache architecture, where multiple data lookups within a single execution frame are grouped together and dispatched as one request, with the results stored in memory for instant retrieval on subsequent calls. The utility distinguishes itself through several key capabilities. It supports per-key error handling, allowing partial failures within a batch without rejecting the entire operation. A cache priming mechanism lets developer

t3code is a platform for operating AI coding agents through a web interface to develop and maintain software projects. It functions as an orchestrator that manages AI agent lifecycles and communication, providing a system for provisioning isolated remote development environments and managing secure connections to cloud-hosted resources. The project utilizes an event-sourced backend architecture that processes typed commands into persisted domain events to maintain system state. It employs a communication system based on JSON-RPC over WebSockets to synchronize real-time state and events betwee

F Prime este un framework bazat pe componente, conceput pentru dezvoltarea și implementarea software-ului pentru sisteme embedded și zbor spațial. Acesta oferă o arhitectură modulară care decuplează logica software de interfețele de comunicare, permițând dezvoltatorilor să definească structurile sistemului printr-un limbaj de modelare specific domeniului. Această abordare bazată pe modele permite generarea automată de cod, asigurând consistența în topologii complexe de sistem, menținând în același timp contracte stricte de interfață între modulele software. Framework-ul se distinge prin sistemul său integrat de build și suita de operațiuni cu date la sol. Acesta automatizează întregul ciclu de viață al software-ului embedded, de la compilarea încrucișată și gestionarea dependențelor până la generarea interfețelor de telemetrie și comandă. Prin furnizarea unui mediu unificat atât pentru software-ul de zbor de la bord, cât și pentru monitorizarea la sol, facilitează integrarea, testarea și controlul sistemelor embedded distribuite pe diverse platforme hardware. Dincolo de arhitectura sa de bază, proiectul include instrumente cuprinzătoare pentru observabilitatea sistemului, inclusiv vizualizarea telemetriei în timp real, logarea evenimentelor și trasarea diagnosticelor. Acesta suportă o gamă largă de scenarii de implementare, de la medii bare-metal la sisteme de operare în timp real, și oferă mecanisme pentru gestionarea memoriei, modelarea comportamentului bazat pe stări și execuția asincronă a sarcinilor. Proiectul este menținut ca un depozit C++ cu documentație extinsă și suport pentru sistemul de build pentru dezvoltare multi-platformă.

OkHttp-OkGo is a network request framework that wraps OkHttp to simplify the process of sending and receiving data from web services. It provides a structured system for managing asynchronous data streams and concurrency through integration with RxJava. The project includes dedicated managers for batch file transfers, enabling the simultaneous upload and download of multiple files. The download system supports breakpoint-based management, allowing interrupted transfers to be paused and resumed from the last saved position. Additional capabilities include a custom HTTP cache manager for defin

Easy Diffusion is a desktop application that generates images from text descriptions using AI. It provides a straightforward interface for creating visuals by simply typing what you want to see, with the ability to preview images as they are being generated. The application supports loading custom AI models, allowing users to switch between different artistic capabilities and styles. It includes tools for editing existing images through text prompts or masks, applying predefined artistic styles like "Realistic" or "Pencil Sketch", and upscaling or correcting facial details after generation. F

This project is a Go language library that provides a programmatic interface for interacting with the Kubernetes API server. It serves as a client for managing cluster resources, offering both typed interfaces for compile-time safety and dynamic interfaces for unstructured data and custom resource management. The library includes a controller framework designed for building event-driven automation. This framework utilizes informers to maintain local resource caches and rate-limited work queues to decouple event detection from state reconciliation. High availability is supported through a lead

Agenda is a persistent background job scheduler and distributed task runner for Node.js applications. It functions as a cron job manager and task queue that ensures background processes survive application restarts by storing job state and metadata in a database. The system coordinates execution across multiple worker instances using distributed locking mechanisms to prevent duplicate processing. It supports flexible scheduling via cron expressions or specific dates and includes a pluggable storage interface for backends such as MongoDB, PostgreSQL, and Redis. The platform provides controls

Resque is a Ruby library for enqueueing and processing asynchronous tasks using Redis as a data store. It functions as a distributed task processor and queue manager, allowing long-running work to be moved out of the main request cycle. The system executes background jobs in isolated child processes to prevent memory leaks and provides a web-based dashboard for monitoring queue depths, worker activity, and failed job statistics. Capability areas include distributed worker coordination via signals, error handling with job retry mechanisms, and priority-ordered queue management. It also suppor

Kue is a Redis-backed job queue library for Node.js that provides a complete system for defining, scheduling, and processing background work. It stores job metadata and state in Redis lists and sorted sets, enabling persistent, in-memory operations with configurable concurrency control and priority-sorted processing. The library includes a RESTful HTTP API for managing jobs and a web-based monitoring dashboard for inspecting job status, progress, and logs. The system distinguishes itself through its event-driven worker model, where workers listen for job events via Redis pub/sub and process j

UFO is a multi-device task orchestrator and LLM agent orchestration framework designed to decompose natural language requests into executable task graphs. It functions as a cross-platform UI automation tool capable of performing interactions on Windows and mobile devices while routing tasks to distributed agents based on their hardware and software capabilities. The system is distinguished by its RAG-enhanced agent architecture, which integrates external documentation and previous execution traces to improve decision-making. It employs a hybrid UI detection approach that combines computer vis

TikTokDownload is a configurable batch video downloader for TikTok and Douyin that strips watermarks and supports automated downloads from user profiles, likes, and collections. It functions as a social media content archiving tool, enabling users to download videos and audio from these platforms for offline viewing or personal backup. The project distinguishes itself through a modular download pipeline that combines audio extraction, batch scheduling, config-driven workflows, cookie-based authentication, URL parsing, paginated API scraping, and watermark removal. It uses a settings file to c

RoadRunner is a high-performance application server and process manager designed to serve PHP applications using a persistent worker model. It eliminates bootload overhead and initialization time by keeping application processes alive between requests, acting as a protocol-agnostic proxy that routes traffic to a pool of supervised workers. The server is built with a plugin-based modular architecture, allowing it to be extended with custom Go plugins and compiled into tailored binaries. It distinguishes itself by providing a unified execution model for a wide array of communication protocols,

This project is a remote file transfer server and asynchronous download orchestrator. It functions as a backend service that manages a queue of remote file transfer requests to coordinate the download of files from the web to a remote storage destination. The system operates as a distributed download coordinator, scheduling and tracking the progress of file transfers across different network nodes. It routes data requests from a client to a remote file system and utilizes asynchronous task execution to process heavy file operations in the background. The server includes capabilities for queu

Machinery is a distributed task queue and asynchronous workflow engine. It provides a system for processing heavy workloads outside the main request flow using a network of distributed background workers and a message-based job orchestrator. The project manages complex task lifecycles through sequential chaining, where results are passed between tasks, and parallel coordination, which can trigger callback tasks upon the completion of a group. It supports periodic workflow scheduling for recurring jobs and delayed execution via specific timestamps. The system includes capabilities for result