18 repositorios
Background processing components that consume and execute tasks from message queues.
Distinguishing note: Focuses on the infrastructure component for asynchronous task processing rather than the message broker itself.
Explore 18 awesome GitHub repositories matching devops & infrastructure · Message Queue Workers. Refine with filters or upvote what's useful.
OpenFaaS is a serverless function platform that provides a container-native framework for deploying and managing event-driven code. It functions as an abstraction layer over container orchestrators, allowing developers to package code into scalable functions that run across Kubernetes clusters or edge computing environments. The platform distinguishes itself through a developer-centric runtime that utilizes standardized language templates and automated build pipelines to simplify the creation of container images. It features a central API gateway that manages request routing, authentication,
In this guide, you'll learn how to deploy from GitLab CI/CD using OpenFaaS's IAM support and Web Identity Federation. You'll need to create YAML files for an Issuer, a Policy and a Role. These need to be applied through
Postal is an open-source mail delivery platform designed for self-hosted sending and receiving of email using standard mail transfer protocols. It serves as a comprehensive email infrastructure that provides full delivery tracking and management of outgoing mail streams. The system includes an SMTP server manager with a web administration interface for configuring server settings and monitoring delivery status. It distinguishes its outbound traffic management through priority-based IP rotation and assignment to help maintain sender reputation. The platform covers incoming email processing an
Uses background worker processes to decouple email reception from final delivery via asynchronous queuing.
This project is a PHP e-commerce platform and enterprise storefront framework designed for building and managing online stores. It functions as a modular PHP application and headless commerce engine, separating e-commerce business logic from frontend presentation to support multiple storefronts and devices. The system is built on a modular architecture that allows developers to add custom business logic and third-party integrations. It utilizes a service-contract-based API to ensure stability across module implementations and employs dependency-injection for object lifecycle management. Core
Provides background workers that consume tasks from message queues to handle resource-intensive workloads asynchronously.
Gorush is a dedicated notification dispatch server used to send real-time alerts to iOS and Android mobile devices. It functions as a centralized infrastructure for hosting a notification API and managing the delivery of push messages through high-throughput queuing. The system supports both gRPC streams and HTTP endpoints to receive notification triggers and includes a command-line interface for triggering alerts without a running server process. It manages network traffic through proxy routing via HTTP or SOCKS5 and secures server communication using automated TLS certificate management. O
Processes high-volume notification deliveries using a configurable pool of concurrent background workers.
Simplebank is a financial services backend application built with Go that manages bank accounts and transfers. It utilizes a dual-protocol interface, providing both gRPC and REST APIs via Protocol Buffers to support different client communication requirements. The system implements a PostgreSQL data layer with versioned schema migrations and type-safe query generation. It handles financial operations through atomic fund transfers and balance change tracking to maintain consistent audit trails. The architecture includes an asynchronous task worker system using a message queue to offload long-
Offloads time-consuming background tasks to a separate worker system using a message queue.
MISP is an open-source threat intelligence sharing platform designed for collecting, storing, and distributing structured threat indicators and intelligence. At its core, it provides a distributed synchronization protocol for transferring events between instances, an attribute-based correlation engine that links matching indicators across events, and a REST API with an OpenAPI specification for programmatic access to threat data. The platform uses formal data formats for JSON, taxonomy, galaxy, and object templates to enable compatibility across tools and communities. The platform distinguish
Processes scheduled tasks and data operations asynchronously through a dedicated background worker queue system.
This is a distributed voting application designed to demonstrate a multi-service architecture. It uses stateless web frontends for submitting votes and viewing live results, with a Redis-backed queue to buffer incoming votes and a PostgreSQL database for persistent tallying. The application is built around asynchronous message queue processing, decoupling the vote submission from the tallying workflow. The project showcases how to deploy a multi-service application using container orchestration tools. It provides YAML-driven declarative deployment manifests for Docker Compose, Docker Swarm, a
Stores incoming votes in a Redis list for asynchronous consumption by a background worker.
protoactor-go is a framework for building concurrent and distributed systems in Go using the actor model. It provides a distributed actor system that enables isolated entities to communicate via asynchronous messaging and share state across a cluster. The framework implements a multi-language actor protocol, allowing interoperability between actors written in Go, C#, and Java. It further supports a virtual actor implementation, where actors are automatically instantiated across a network based on a unique identity. The system includes a supervision model for managing actor lifecycles and fau
Distributes incoming messages across pools of worker actors to balance load and increase throughput.
Flask-SocketIO is a Flask extension that integrates the Socket.IO real-time communication protocol into Flask applications, enabling persistent bidirectional connections between servers and clients. It transparently switches between WebSocket and HTTP long-polling while maintaining a persistent session identity, and provides a full event-driven framework for real-time web application communication. The library distinguishes itself with support for namespace multiplexing, allowing event handlers to be isolated into separate namespaces with independent lifecycle and routing. It also offers room
Uses a shared message queue to synchronize events across multiple worker processes for horizontal scaling.
Dramatiq is a distributed task queue and workload manager used to offload function execution to background workers. It functions as an asynchronous task orchestrator that enables the distribution of computational tasks across a cluster using a pluggable transport layer supporting RabbitMQ and Redis. The framework provides specialized tools for complex task orchestration, including the ability to link background jobs into sequences, pipelines, and barriers. It further manages distributed concurrency through the use of shared mutexes, rate limiters, and exponential backoff retries to prevent re
Provides background processing components that consume and execute tasks from declared message queues.
Ockam es un framework de cifrado end-to-end y proveedor de identidad distribuida diseñado para establecer comunicación segura entre aplicaciones y dispositivos. Proporciona una superposición de red segura que utiliza identidades criptográficas y control de acceso basado en atributos para implementar el acceso a la red de confianza cero (zero trust). El proyecto se distingue por el enrutamiento de múltiples saltos impulsado por metadatos y una capa de transporte conectable, permitiendo que el tráfico cifrado se mueva a través de diversas topologías de red sin requerir superposiciones de IP virtuales. Permite específicamente el túnel seguro para aplicaciones heredadas envolviendo el tráfico TCP crudo en canales cifrados, permitiendo la conectividad de red privada y el bypass de firewall a través de relés de salida. La plataforma cubre una amplia gama de capacidades, incluyendo gestión de identidad distribuida, emisión y verificación de credenciales criptográficas y la ejecución de actores concurrentes con estado. También proporciona herramientas para el aprovisionamiento de nodos a escala de nube y despliegue automatizado utilizando plantillas de infraestructura como código.
Directs communication between decoupled actor components using a registry to resolve addresses and relay messages.
Ockam es un framework de redes de confianza cero diseñado para asegurar el tránsito de datos entre aplicaciones distribuidas utilizando una superposición de red basada en identidad. Proporciona las primitivas necesarias para establecer conexiones autenticadas mutuamente y cifradas de extremo a extremo, eliminando la dependencia de la seguridad tradicional de la capa de red. El proyecto se distingue por su uso de control de acceso basado en atributos y credenciales verificables para gestionar la confianza a escala. Implementa la rotación de identidad criptográfica para mantener la continuidad de la identidad y se integra con sistemas de gestión de claves respaldados por hardware para asegurar claves privadas dentro de enclaves o servicios de gestión de claves en la nube. La plataforma cubre una amplia gama de capacidades, incluyendo enrutamiento binario de múltiples saltos y puente de red basado en relés para conectar redes dispares. Puede envolver tráfico TCP o Kafka heredado en túneles seguros, permitiendo que los servicios privados se comuniquen sin exponer puertos de escucha. Además, emplea un modelo de actor con estado para procesar mensajes de forma asíncrona a través de nodos distribuidos. El despliegue es compatible a través de plantillas de infraestructura como código para el aprovisionamiento de nodos seguros y pasarelas en entornos de nube.
Sends data to specific worker addresses through a router that resolves locations and manages delivery across different contexts.
useSend is an email marketing platform and transactional delivery system built to manage bulk campaigns and individual messages using AWS SES for delivery. It provides a RESTful API, an SMTP relay gateway, and a visual management interface for designing HTML templates and coordinating email broadcasts. The platform distinguishes itself by translating legacy SMTP traffic into modern API calls and implementing an event-driven webhook system. This system pushes real-time delivery, bounce, and click notifications to external servers using HMAC-SHA256 signed payloads to ensure authenticity. The s
Processes bulk email campaigns and scheduled messages using background workers for high throughput.
This project is a blockchain node software implementation that maintains a decentralized ledger using the Ouroboros proof-of-stake consensus protocol. It provides a complete environment for operating a distributed record, including a cryptocurrency wallet backend and a peer-to-peer network layer. The system is distinguished by its use of a UTXO-based ledger model and a diffusion-based state synchronization layer for node communication. It integrates a Public Key Infrastructure with TLS-authenticated communication to secure inter-node traffic and employs hierarchical deterministic key derivati
Coordinates one-way and bi-directional communication between node components using event dispatching threads.
This project is a headless WhatsApp client and automation server that exposes multi-device functionality as a REST API gateway. It provides a centralized interface for managing multiple account sessions, automating messages, and developing bots through a standardized protocol. The system distinguishes itself by providing a secure webhook provider that forwards real-time messages and reactions to external URLs using signature verification. It includes specialized media utilities for converting images and GIFs into the WebP format required for stickers, as well as tools to simulate active user
Uses Go routines to process incoming events and outgoing messages without blocking the main execution thread.
The Cloudflare Workers Rust SDK is a toolchain and framework for building high-performance serverless functions using Rust. It compiles code into WebAssembly modules that execute within a sandboxed host environment at the edge. The project provides a typed remote procedure call system for invoking functions across different worker instances and utilizes procedural macros for mapping these distributed interfaces. It uses trait-based bindings and standardized wrappers to provide type-safe access to platform services and web-standard request and response objects. The SDK covers a broad range of
Implements background processing components that consume and execute tasks from message queues.
S3Scanner is a security tool for auditing public access and misconfigured permissions across S3-compatible storage providers. It functions as a storage auditor and security scanner designed to identify open buckets, enumerate publicly accessible objects, and exfiltrate data from misconfigured cloud environments. The project is distinguished by its integration with message brokers, allowing it to consume target lists for large-scale cloud infrastructure audits. It also provides utilities for dumping the entire contents of misconfigured buckets to local directories and calculating the total siz
Uses background workers to consume targets from message queues for distributed high-volume scanning.
This project is a high-performance PHP framework designed for building scalable services using a resident memory execution model. By maintaining application state in memory across requests, it eliminates bootstrap overhead and provides a persistent environment for long-running processes. The framework distinguishes itself through its integrated support for durable workflow orchestration and asynchronous task management. It enables the execution of complex, long-running business processes that automatically handle failures and state recovery, alongside a background processing system that dispa
Dispatches asynchronous jobs to background processes using supervised message brokers to maintain responsiveness.