Projects sharing features with Celery

Hangfire is a background job scheduler and distributed task queue for .NET applications. It serves as a job orchestration framework that offloads heavy processing to background workers using a SQL-backed processor to manage job state across multiple servers. The framework distinguishes itself through reliable task scheduling, where job metadata and arguments are persisted in an external database to ensure tasks survive application restarts. It supports advanced orchestration patterns, including the ability to chain dependent tasks so that a child job triggers automatically upon the successful

Dramatiq is a distributed task queue and workload manager used to offload function execution to background workers. It functions as an asynchronous task orchestrator that enables the distribution of computational tasks across a cluster using a pluggable transport layer supporting RabbitMQ and Redis. The framework provides specialized tools for complex task orchestration, including the ability to link background jobs into sequences, pipelines, and barriers. It further manages distributed concurrency through the use of shared mutexes, rate limiters, and exponential backoff retries to prevent re

RoadRunner is a high-performance application server and process manager designed to serve PHP applications using a persistent worker model. It eliminates bootload overhead and initialization time by keeping application processes alive between requests, acting as a protocol-agnostic proxy that routes traffic to a pool of supervised workers. The server is built with a plugin-based modular architecture, allowing it to be extended with custom Go plugins and compiled into tailored binaries. It distinguishes itself by providing a unified execution model for a wide array of communication protocols,

Machinery is a distributed task queue and asynchronous workflow engine. It provides a system for processing heavy workloads outside the main request flow using a network of distributed background workers and a message-based job orchestrator. The project manages complex task lifecycles through sequential chaining, where results are passed between tasks, and parallel coordination, which can trigger callback tasks upon the completion of a group. It supports periodic workflow scheduling for recurring jobs and delayed execution via specific timestamps. The system includes capabilities for result

Flower is a monitoring and administration tool for Celery task queues. It provides a real-time web dashboard and a REST API to monitor distributed task clusters, manage worker instances, and observe message broker health. The project distinguishes itself by offering centralized control over the task lifecycle, allowing users to trigger, revoke, or terminate tasks and apply execution rate limits. It also includes a Prometheus metrics exporter to surface internal performance and status data for external monitoring and alerting systems. The tool covers a broad range of observability and managem

RocketMQ is a cloud-native distributed messaging platform and streaming engine. It functions as a distributed transactional queue that ensures atomicity between local transactions and message delivery, and serves as an MQTT IoT message broker to bridge lightweight device traffic into high-performance data streams. The system is distinguished by a Kubernetes-native architecture that decouples compute from storage to allow independent scaling of traffic and data retention. It utilizes a tiered storage model to offload older data to remote storage and employs quorum-based replication and automat

Hyperf is a high-performance PHP coroutine framework designed for building microservices and middleware. It utilizes non-blocking coroutines to handle high concurrency and low-latency request processing, providing a foundation for scalable distributed systems. The framework is distinguished by an aspect-oriented programming based dependency injector that enables pluggable components and meta-programming. It includes a coroutine-optimized object-relational mapper with integrated model caching and an orchestration toolkit for microservice governance, featuring service discovery, circuit breaker

NSQ is a distributed, brokerless messaging platform designed for high-throughput, fault-tolerant communication. By utilizing a decentralized topology, it eliminates single points of failure and allows for horizontal scaling across clusters. The system organizes message streams into topics and channels, effectively decoupling producers from consumers to support both streaming and job-oriented workloads. The platform distinguishes itself through a lookup-service-based discovery mechanism that enables clients to dynamically locate producers at runtime without requiring centralized coordination.

This project is a self-hosted email verification system and API designed to validate email existence and clean mailing lists on private infrastructure. It functions as a deliverability tool that confirms if email addresses are reachable by communicating with mail servers via the SMTP protocol without sending actual messages. The system is distinguished by its high-volume SMTP infrastructure, which utilizes a stateless worker architecture and message queue task distribution to scale validation tasks. It includes an SMTP proxy gateway that routes requests through SOCKS5 proxies to mask server i

NATS Server is a high-performance, lightweight messaging system designed for cloud-native applications, edge computing, and distributed microservices. It functions as a distributed publish-subscribe broker that routes messages using hierarchical, dot-separated subject strings, enabling decoupled communication between services without requiring centralized broker lookups. The system supports core messaging patterns including asynchronous publish-subscribe, request-reply, and load-balanced queue processing. The platform distinguishes itself through a decentralized architecture that eliminates t

This project is a comprehensive educational resource and study guide focused on distributed systems architecture and backend infrastructure design. It provides a structured curriculum for mastering the principles of scalability, reliability, and performance required to design complex software systems. The repository distinguishes itself by offering a methodical approach to technical interview preparation, incorporating design patterns, architectural trade-offs, and spaced repetition tools to help users retain complex concepts. It emphasizes constraint-driven analysis, teaching users how to ev

Faktory is an open-source work server that queues, dispatches, and manages background jobs across multiple programming languages. It stores job payloads as JSON hashes in a Redis-backed queue and provides language-specific client and worker libraries that enable any language to push jobs to the server or fetch and execute them. The server includes a batch workflow orchestrator that groups jobs into batches with completion tracking for coordinating multi-step asynchronous workflows. It features a configurable job uniqueness filter that prevents duplicate enqueues within a time window, an expon

Flyte is a Kubernetes-based machine learning orchestrator and containerized pipeline manager designed for coordinating AI workflows and data pipelines. It functions as an engine for defining and executing resilient pipelines, utilizing a data lineage tracker to maintain immutable execution states and ensure reproducible outputs. The platform distinguishes itself by packaging individual tasks into separate containers to ensure dependency isolation and environment consistency. It provides specialized capabilities for machine learning, including the transformation of trained models into scalable

This project provides a framework for AI agent orchestration and context management, enabling the deployment of specialized AI personas and subagents to solve multi-step technical goals. It centers on managing specialized agents with isolated contexts and role-based prompts to handle domain-specific tasks. The system differentiates itself through a hierarchical project memory using markdown files to maintain coding standards and a secure execution model that utilizes sandboxed environments and git worktree isolation. It also features a Model Context Protocol integration for external tool conn

This project provides a structured curriculum and visual guide for mastering web development within the ASP.NET Core ecosystem. It serves as a comprehensive roadmap that maps out the essential technologies, milestones, and proficiency sequences required for developers to progress from beginner to advanced levels. The repository distinguishes itself by curating high-quality learning resources and technical documentation into a logical progression. It visualizes complex development paths through structured diagrams, helping users navigate the technical requirements of building and maintaining m

Beanstalkd is a distributed work queue server designed for asynchronous job processing. It functions as a standalone server that distributes background tasks between producers and consumers to improve application responsiveness and throughput. The system organizes tasks using numeric priority levels to ensure critical work is processed first. It manages the job lifecycle through discrete states and uses a simple line-based text protocol over TCP for communication. To ensure reliability, the server persists job data to a sequential disk log, allowing the queue state to be recovered after a sy

Olmocr is a distributed document processing framework designed to convert PDF and image files into structured markdown. It functions as a vision-based document parser that utilizes multimodal neural networks to interpret complex visual layouts and translate them into standardized text representations. The system operates as a remote inference orchestrator, offloading heavy document analysis tasks to external servers or cloud APIs to minimize local computational requirements. By employing a stateless worker architecture, it decouples document ingestion from inference, allowing for the distribu

Temporal is a distributed workflow orchestration engine designed to manage fault-tolerant, stateful, and long-running background processes. It functions as a platform for coordinating complex cross-service operations, ensuring consistency and reliability in distributed environments by decoupling workflow orchestration from task execution. The platform distinguishes itself through a deterministic, event-sourced execution model that reconstructs workflow state by re-executing code from an immutable event log. This approach isolates non-deterministic side effects into managed activities, allowin

Resque is a Ruby library for enqueueing and processing asynchronous tasks using Redis as a data store. It functions as a distributed task processor and queue manager, allowing long-running work to be moved out of the main request cycle. The system executes background jobs in isolated child processes to prevent memory leaks and provides a web-based dashboard for monitoring queue depths, worker activity, and failed job statistics. Capability areas include distributed worker coordination via signals, error handling with job retry mechanisms, and priority-ordered queue management. It also suppor

ClusterFuzz is an automated platform that runs coverage-guided fuzzers at scale to find security and stability bugs in software. It orchestrates libFuzzer and AFL++ across distributed clusters of worker bots, collecting coverage feedback to guide input mutation and discover crashes. The platform provides a web-based dashboard for configuring fuzzing jobs, monitoring progress, and inspecting crash reports, with role-based access control to restrict sensitive features. The system automates the full fuzzing lifecycle, from build pipeline integration and corpus management to crash triage and bug

Goravel is a full-featured development scaffold and framework for building web applications, REST APIs, and gRPC services using the Go programming language. It implements a model-view-controller architecture and provides a comprehensive toolkit for high-performance remote procedure call servers and clients. The framework is distinguished by its extensive integrated ecosystem, which includes a fluent object-relational mapper for database management and a dedicated command-line interface toolkit for administrative automation and project scaffolding. It features a driver-based service abstractio

Sidekiq is a Ruby background processing framework and asynchronous task runner. It functions as a Redis-backed background job processor that offloads heavy or time-consuming work from web requests to separate worker processes to ensure the main application remains responsive. The system operates as a Redis task queue, storing pending jobs in Redis to be processed concurrently by multiple threads. It provides a framework for distributed task queueing and asynchronous job scheduling to coordinate work across multiple server instances. The project covers Ruby application scaling by executing ba

This is a Golang client library for interacting with a cloud native distributed messaging system. It provides the necessary tools for Go applications to exchange messages using publish-subscribe and request-reply patterns, as well as specialized clients for managing persistent streams and distributed storage. The library includes a JetStream client for durable message streaming and replay, a Key-Value store client for managing distributed state with versioning and watchers, and an Object Store client for the storage and retrieval of large binary files via chunked delivery. The implementation

cookiecutter-django is a template-based project generator and production-ready scaffold for Django web applications. It functions as a boilerplate that injects user-defined variables into predefined file templates to automate the creation of a standardized directory structure and initial project configuration. The project provides a production blueprint that integrates a customizable user authentication system, environment-variable configuration, and a containerized development environment. It bundles Django with databases and task queues to ensure consistency across local and production work

This project provides a collection of Helm charts designed for deploying and managing self-hosted error tracking and performance monitoring infrastructure within a Kubernetes cluster. It enables the deployment of complex, multi-service application stacks using standardized configuration templates to ensure consistent and repeatable infrastructure management. The platform distinguishes itself by automating the operational requirements of distributed event processing, including the horizontal autoscaling of background task workers and the secure generation of cryptographic keys for internal ser

This project provides a containerized environment for deploying Apache Airflow, enabling the orchestration of complex data pipelines and automated task scheduling. By packaging the orchestration platform into portable images, it ensures consistent execution across diverse infrastructure setups and simplifies the management of runtime dependencies. The platform facilitates distributed task execution by decoupling the scheduler from the execution layer, allowing for horizontal scaling of processing power across multiple worker nodes. It supports dynamic configuration through environment variabl