Awesome GitHub RepositoriesBackground Job Queues

Mastodon is a self-hosted, decentralized social networking platform that functions as a microblogging application. It enables independent server instances to communicate and exchange social data through the standardized ActivityPub protocol, allowing users to participate in a global, interoperable network. The platform distinguishes itself through its federated architecture, which grants administrators full control over their community instances. This includes comprehensive tools for user moderation, account management, and the enforcement of community guidelines. The system is designed to ha

Payload is a headless content management system and application framework that uses a code-first approach to define data schemas and administrative interfaces. By utilizing a centralized, type-safe configuration object, it automatically generates database schemas, API endpoints, and a fully customizable admin panel. The system is built on a database-agnostic architecture, allowing it to interface with various storage engines while providing a unified, type-safe API for server-side operations, REST, and GraphQL. What distinguishes Payload is its deep extensibility and developer-centric design.

This project is a full-stack web framework that provides a comprehensive environment for building server-side applications. It utilizes a model-view-controller architecture to separate application logic into distinct layers for data management, user interface presentation, and request handling. The platform manages the entire request-response lifecycle, including security, session handling, and background task processing, while using an object-relational mapping layer to translate database records into programmable objects. The framework distinguishes itself through a central service containe

Vapor is a comprehensive server-side web framework designed for building scalable, high-performance applications and APIs in Swift. It provides a non-blocking, event-loop-based runtime that manages concurrent task processing, background job queues, and asynchronous request handling. The framework is built around a dependency injection container that manages the lifecycle and resolution of services, configurations, and database connections throughout the request pipeline. The framework distinguishes itself through a protocol-oriented design that emphasizes type safety across all layers of the

Devise is a flexible authentication framework for Ruby on Rails applications. It serves as a user identity management system that handles registration, sign-in, password recovery, and account confirmation. The system integrates with the Warden middleware to manage user sessions and security across various request types. The framework functions as a multi-model authentication engine, allowing for the configuration of multiple distinct user models with independent routes and access controls. It also provides a standardized interface to connect with external identity providers and third-party lo

Devise is a comprehensive identity management system and authentication framework for Ruby on Rails applications. It provides a complete set of tools for managing user registration, secure sign-in, and session handling using a modular strategy pattern. The framework distinguishes itself by offering a suite of security hardening features, including brute force protection through account locking and secure password recovery workflows. It also functions as an integrator for external identity providers and third-party authentication via standardized protocols. Broad capabilities cover the full u

Excelize is a library for reading and writing spreadsheet files in the Office Open XML format. It provides a comprehensive suite of tools for programmatically creating, modifying, and analyzing workbooks, worksheets, and cell data, ensuring compatibility across various office software suites through structured XML serialization. The library distinguishes itself with a built-in formula calculation engine that evaluates complex mathematical and logical expressions directly against workbook data. It also features a memory-mapped streaming architecture, which allows for the efficient processing o

Diaspora is a federated social networking platform that allows users to run and manage self-hosted community servers, known as pods. It operates as a distributed network where independent server nodes exchange content and users using open protocols and standardized communication schemas. The platform is distinguished by its focus on decentralized identity management and privacy-preserving communication. It includes a privacy-focused media proxy that routes external assets through a local server to protect user identity and supports cross-instance account migration, allowing users to move thei

Sidekiq is a background job processor and queue manager for Ruby that uses Redis to manage asynchronous tasks. It functions as a distributed task scheduler capable of handling periodic, delayed, and recurring jobs across a cluster of worker processes. The project features a job monitoring dashboard and administrative web interface for visualizing system state, tracking worker performance, and managing failed or dead jobs. It provides a distributed rate limiter to control execution frequency across multiple processes. The framework covers a broad range of operational capabilities, including j

Laravel-Excel is an integration library for importing and exporting spreadsheet data between Laravel applications and Excel or CSV files. It provides a suite of tools for bidirectional spreadsheet integration, including a system for reading workbooks and mapping data into database models. The library distinguishes itself through a background processing system that handles large imports and exports using chunking and job queues. It supports template-driven exports by converting HTML tables from view templates into spreadsheet cells. The toolset covers broad capabilities for large dataset proc

This is a Laravel Excel integration library used for importing and exporting spreadsheet files within the Laravel framework. It functions as both a spreadsheet data exporter for converting database queries and collections into files and a spreadsheet data importer for mapping workbook data to database models. The library includes an HTML to spreadsheet converter that transforms HTML table templates into downloadable documents to preserve specific layout designs. It supports large dataset exports via background queuing and processes bulk data imports through chunked reading and batch inserts.

Misskey is a self-hosted, decentralized microblogging platform and federated social media server. It functions as a distributed content management system that allows users to communicate across multiple independent and interconnected server instances using the ActivityPub protocol. The platform distinguishes itself with a dynamic application engine that allows for the creation of interactive applications and custom page layouts using a scripting language. It also features a specialized markup language for rich text rendering, enabling the use of animations and custom styles for consistent con

rq is a distributed task queue and background worker system for Python that uses a Redis backend to decouple task submission from execution. It functions as a reliable message queue and task scheduler, allowing Python functions or asyncio coroutines to be processed asynchronously across multiple worker processes. The project distinguishes itself through reliable queuing mechanisms that prevent job loss during worker crashes using atomic operations. It provides specialized orchestration capabilities, including the prevention of duplicate jobs, job execution prioritization, and the ability to m

Hangfire is a background job scheduler and distributed task queue for .NET applications. It serves as a job orchestration framework that offloads heavy processing to background workers using a SQL-backed processor to manage job state across multiple servers. The framework distinguishes itself through reliable task scheduling, where job metadata and arguments are persisted in an external database to ensure tasks survive application restarts. It supports advanced orchestration patterns, including the ability to chain dependent tasks so that a child job triggers automatically upon the successful

PyCaret is a Python AutoML platform and MLOps lifecycle manager designed to automate machine learning workflows. It functions as a low-code environment that leverages a scikit-learn native engine to execute preprocessing, training, and evaluation for tabular data. The platform distinguishes itself as an LLM-powered ML copilot, using large language model agents to analyze datasets, design experiment configurations, and explain model results. It also serves as a Kubernetes ML orchestrator and model registry, enabling the versioning of trained pipelines and their promotion to production API endp

Resque is a Ruby library for enqueueing and processing asynchronous tasks using Redis as a data store. It functions as a distributed task processor and queue manager, allowing long-running work to be moved out of the main request cycle. The system executes background jobs in isolated child processes to prevent memory leaks and provides a web-based dashboard for monitoring queue depths, worker activity, and failed job statistics. Capability areas include distributed worker coordination via signals, error handling with job retry mechanisms, and priority-ordered queue management. It also suppor

RoadRunner is a high-performance application server and process manager designed to serve PHP applications using a persistent worker model. It eliminates bootload overhead and initialization time by keeping application processes alive between requests, acting as a protocol-agnostic proxy that routes traffic to a pool of supervised workers. The server is built with a plugin-based modular architecture, allowing it to be extended with custom Go plugins and compiled into tailored binaries. It distinguishes itself by providing a unified execution model for a wide array of communication protocols,

Buffalo este un framework web MVC cuprinzător și un toolchain full-stack pentru construirea aplicațiilor web în Go. Oferă un mediu de dezvoltare structurat care separă modelele, view-urile și controllerele, integrând un framework web cu un wrapper dedicat de tip object-relational mapper (ORM) pentru gestionarea schemei bazei de date și a înregistrărilor. Proiectul se distinge printr-un set de utilitare de dezvoltare pentru bootstrapping-ul structurilor de proiect, gestionarea activelor și construirea de containere gata de producție. Dispune de un sistem de rutare sofisticat care suportă generarea de resurse RESTful bazată pe convenții, găzduire virtuală bazată pe domeniu și crearea de route helpers pentru link-uri interne consistente. Framework-ul acoperă o gamă largă de capabilități web de bază, inclusiv pipeline-uri de cereri bazate pe middleware, randarea dinamică a șabloanelor și gestionarea stării la nivel de cerere. Include, de asemenea, suport integrat pentru procesarea sarcinilor în fundal, gestionarea sesiunilor, trimiterea de e-mailuri și închiderea grațioasă a serverului.

BullMQ is a Redis-backed message queue library and background processor designed for distributed task queueing. It functions as a distributed queue manager and task scheduler, utilizing Redis to manage asynchronous job processing and persistence. The system distinguishes itself through its role as a job workflow orchestrator, enabling the definition of complex parent-child job dependencies and hierarchies for multi-step workflows. It provides sandboxed process execution to isolate heavy workloads and prevent event loop blocking, alongside distributed rate limiting to protect downstream servic

Machinery is a distributed task queue and asynchronous workflow engine. It provides a system for processing heavy workloads outside the main request flow using a network of distributed background workers and a message-based job orchestrator. The project manages complex task lifecycles through sequential chaining, where results are passed between tasks, and parallel coordination, which can trigger callback tasks upon the completion of a group. It supports periodic workflow scheduling for recurring jobs and delayed execution via specific timestamps. The system includes capabilities for result