Awesome GitHub RepositoriesTask Queues

Paperless-ngx is a self-hosted document management server designed to transform physical paperwork into a searchable, organized digital archive. It functions as a private platform for storing, indexing, and retrieving documents, providing users with full control over their data on local infrastructure or private cloud servers. The system distinguishes itself through an automated workflow engine that categorizes, tags, and routes incoming files using content analysis and metadata extraction. To maintain responsiveness during resource-intensive tasks like optical character recognition, it utili

Seal is a mobile application designed to retrieve video and audio content from various online platforms. It functions as a graphical interface that manages background transfer processes, allowing users to download and archive media files directly to their local device storage for offline access. The application distinguishes itself by acting as a bridge to powerful command-line utilities, orchestrating these external binaries to handle complex media extraction and file conversion tasks. Users can customize their experience through a declarative template system that defines specific execution

ArchiveBox is a self-hosted archiving tool designed for personal digital preservation and research data management. It functions as an automated web preservation engine that monitors URL inputs from bookmarks, browser history, or manual entries to capture and store permanent, offline copies of web content. By utilizing headless browser automation, the system renders dynamic web pages to ensure that captured snapshots, PDFs, and media assets remain accurate and accessible even if the original source disappears. The project distinguishes itself through a modular extractor pipeline and a task-qu

Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob

This project provides a framework for managing multi-agent systems, designed to automate complex software development, infrastructure, and business workflows. It functions as a multi-agent workflow orchestrator that routes tasks to domain-specific workers while maintaining state persistence and infrastructure automation. By leveraging large language models, the system decomposes high-level objectives into actionable plans, ensuring that complex operations are executed with consistency and reliability. The framework distinguishes itself through its hierarchical agent registry and policy-driven

Temporal is a distributed workflow orchestration engine designed to manage fault-tolerant, stateful, and long-running background processes. It functions as a platform for coordinating complex cross-service operations, ensuring consistency and reliability in distributed environments by decoupling workflow orchestration from task execution. The platform distinguishes itself through a deterministic, event-sourced execution model that reconstructs workflow state by re-executing code from an immutable event log. This approach isolates non-deterministic side effects into managed activities, allowin

DataLoader is a utility that collects individual data loads into a single batch and caches results to minimize redundant backend requests. It operates on a batch-and-cache architecture, where multiple data lookups within a single execution frame are grouped together and dispatched as one request, with the results stored in memory for instant retrieval on subsequent calls. The utility distinguishes itself through several key capabilities. It supports per-key error handling, allowing partial failures within a batch without rejecting the entire operation. A cache priming mechanism lets developer

t3code is a platform for operating AI coding agents through a web interface to develop and maintain software projects. It functions as an orchestrator that manages AI agent lifecycles and communication, providing a system for provisioning isolated remote development environments and managing secure connections to cloud-hosted resources. The project utilizes an event-sourced backend architecture that processes typed commands into persisted domain events to maintain system state. It employs a communication system based on JSON-RPC over WebSockets to synchronize real-time state and events betwee

F Prime هو إطار عمل قائم على المكونات مصمم لتطوير ونشر البرمجيات المدمجة وبرمجيات الطيران الفضائي. يوفر بنية معيارية تفصل منطق البرمجيات عن واجهات الاتصال، مما يسمح للمطورين بتحديد هياكل النظام من خلال لغة نمذجة خاصة بالمجال. يتيح هذا النهج القائم على النماذج توليد الكود آلياً، مما يضمن الاتساق عبر طوبولوجيا النظام المعقدة مع الحفاظ على عقود واجهة صارمة بين وحدات البرمجيات. يتميز إطار العمل بنظام بناء متكامل ومجموعة عمليات بيانات أرضية. فهو يقوم بأتمتة دورة حياة البرمجيات المدمجة بالكامل، بدءاً من التجميع المتقاطع وإدارة التبعيات وصولاً إلى توليد واجهات القياس عن بُعد والأوامر. من خلال توفير بيئة موحدة لكل من برمجيات الطيران الموجودة على متن المركبة والمراقبة الأرضية، فإنه يسهل التكامل والاختبار والتحكم في الأنظمة المدمجة الموزعة عبر منصات أجهزة متنوعة. بالإضافة إلى بنيته الأساسية، يتضمن المشروع أدوات شاملة لمراقبة النظام، بما في ذلك تصور القياس عن بُعد في الوقت الفعلي، وتسجيل الأحداث، وتتبع التشخيص. وهو يدعم مجموعة واسعة من سيناريوهات النشر، من بيئات الأجهزة الخام (bare-metal) إلى أنظمة التشغيل في الوقت الفعلي، ويوفر آليات لإدارة الذاكرة، ونمذجة السلوك القائم على الحالة، وتنفيذ المهام غير المتزامنة. يتم صيانة المشروع كمستودع C++ مع توثيق شامل ودعم لنظام البناء من أجل التطوير عبر المنصات.

OkHttp-OkGo is a network request framework that wraps OkHttp to simplify the process of sending and receiving data from web services. It provides a structured system for managing asynchronous data streams and concurrency through integration with RxJava. The project includes dedicated managers for batch file transfers, enabling the simultaneous upload and download of multiple files. The download system supports breakpoint-based management, allowing interrupted transfers to be paused and resumed from the last saved position. Additional capabilities include a custom HTTP cache manager for defin

Easy Diffusion is a desktop application that generates images from text descriptions using AI. It provides a straightforward interface for creating visuals by simply typing what you want to see, with the ability to preview images as they are being generated. The application supports loading custom AI models, allowing users to switch between different artistic capabilities and styles. It includes tools for editing existing images through text prompts or masks, applying predefined artistic styles like "Realistic" or "Pencil Sketch", and upscaling or correcting facial details after generation. F

This project is a Go language library that provides a programmatic interface for interacting with the Kubernetes API server. It serves as a client for managing cluster resources, offering both typed interfaces for compile-time safety and dynamic interfaces for unstructured data and custom resource management. The library includes a controller framework designed for building event-driven automation. This framework utilizes informers to maintain local resource caches and rate-limited work queues to decouple event detection from state reconciliation. High availability is supported through a lead

Agenda is a persistent background job scheduler and distributed task runner for Node.js applications. It functions as a cron job manager and task queue that ensures background processes survive application restarts by storing job state and metadata in a database. The system coordinates execution across multiple worker instances using distributed locking mechanisms to prevent duplicate processing. It supports flexible scheduling via cron expressions or specific dates and includes a pluggable storage interface for backends such as MongoDB, PostgreSQL, and Redis. The platform provides controls

Resque is a Ruby library for enqueueing and processing asynchronous tasks using Redis as a data store. It functions as a distributed task processor and queue manager, allowing long-running work to be moved out of the main request cycle. The system executes background jobs in isolated child processes to prevent memory leaks and provides a web-based dashboard for monitoring queue depths, worker activity, and failed job statistics. Capability areas include distributed worker coordination via signals, error handling with job retry mechanisms, and priority-ordered queue management. It also suppor

Kue is a Redis-backed job queue library for Node.js that provides a complete system for defining, scheduling, and processing background work. It stores job metadata and state in Redis lists and sorted sets, enabling persistent, in-memory operations with configurable concurrency control and priority-sorted processing. The library includes a RESTful HTTP API for managing jobs and a web-based monitoring dashboard for inspecting job status, progress, and logs. The system distinguishes itself through its event-driven worker model, where workers listen for job events via Redis pub/sub and process j

UFO is a multi-device task orchestrator and LLM agent orchestration framework designed to decompose natural language requests into executable task graphs. It functions as a cross-platform UI automation tool capable of performing interactions on Windows and mobile devices while routing tasks to distributed agents based on their hardware and software capabilities. The system is distinguished by its RAG-enhanced agent architecture, which integrates external documentation and previous execution traces to improve decision-making. It employs a hybrid UI detection approach that combines computer vis

TikTokDownload is a configurable batch video downloader for TikTok and Douyin that strips watermarks and supports automated downloads from user profiles, likes, and collections. It functions as a social media content archiving tool, enabling users to download videos and audio from these platforms for offline viewing or personal backup. The project distinguishes itself through a modular download pipeline that combines audio extraction, batch scheduling, config-driven workflows, cookie-based authentication, URL parsing, paginated API scraping, and watermark removal. It uses a settings file to c

RoadRunner is a high-performance application server and process manager designed to serve PHP applications using a persistent worker model. It eliminates bootload overhead and initialization time by keeping application processes alive between requests, acting as a protocol-agnostic proxy that routes traffic to a pool of supervised workers. The server is built with a plugin-based modular architecture, allowing it to be extended with custom Go plugins and compiled into tailored binaries. It distinguishes itself by providing a unified execution model for a wide array of communication protocols,

This project is a remote file transfer server and asynchronous download orchestrator. It functions as a backend service that manages a queue of remote file transfer requests to coordinate the download of files from the web to a remote storage destination. The system operates as a distributed download coordinator, scheduling and tracking the progress of file transfers across different network nodes. It routes data requests from a client to a remote file system and utilizes asynchronous task execution to process heavy file operations in the background. The server includes capabilities for queu

Machinery is a distributed task queue and asynchronous workflow engine. It provides a system for processing heavy workloads outside the main request flow using a network of distributed background workers and a message-based job orchestrator. The project manages complex task lifecycles through sequential chaining, where results are passed between tasks, and parallel coordination, which can trigger callback tasks upon the completion of a group. It supports periodic workflow scheduling for recurring jobs and delayed execution via specific timestamps. The system includes capabilities for result