17 个仓库
Background processing components that consume and execute tasks from message queues.
Distinguishing note: Focuses on the infrastructure component for asynchronous task processing rather than the message broker itself.
Explore 17 awesome GitHub repositories matching devops & infrastructure · Message Queue Workers. Refine with filters or upvote what's useful.
OpenFaaS is a serverless function platform that provides a container-native framework for deploying and managing event-driven code. It functions as an abstraction layer over container orchestrators, allowing developers to package code into scalable functions that run across Kubernetes clusters or edge computing environments. The platform distinguishes itself through a developer-centric runtime that utilizes standardized language templates and automated build pipelines to simplify the creation of container images. It features a central API gateway that manages request routing, authentication,
In this guide, you'll learn how to deploy from GitLab CI/CD using OpenFaaS's IAM support and Web Identity Federation. You'll need to create YAML files for an Issuer, a Policy and a Role. These need to be applied through
Postal is an open-source mail delivery platform designed for self-hosted sending and receiving of email using standard mail transfer protocols. It serves as a comprehensive email infrastructure that provides full delivery tracking and management of outgoing mail streams. The system includes an SMTP server manager with a web administration interface for configuring server settings and monitoring delivery status. It distinguishes its outbound traffic management through priority-based IP rotation and assignment to help maintain sender reputation. The platform covers incoming email processing an
Uses background worker processes to decouple email reception from final delivery via asynchronous queuing.
This project is a PHP e-commerce platform and enterprise storefront framework designed for building and managing online stores. It functions as a modular PHP application and headless commerce engine, separating e-commerce business logic from frontend presentation to support multiple storefronts and devices. The system is built on a modular architecture that allows developers to add custom business logic and third-party integrations. It utilizes a service-contract-based API to ensure stability across module implementations and employs dependency-injection for object lifecycle management. Core
Provides background workers that consume tasks from message queues to handle resource-intensive workloads asynchronously.
Gorush is a dedicated notification dispatch server used to send real-time alerts to iOS and Android mobile devices. It functions as a centralized infrastructure for hosting a notification API and managing the delivery of push messages through high-throughput queuing. The system supports both gRPC streams and HTTP endpoints to receive notification triggers and includes a command-line interface for triggering alerts without a running server process. It manages network traffic through proxy routing via HTTP or SOCKS5 and secures server communication using automated TLS certificate management. O
Processes high-volume notification deliveries using a configurable pool of concurrent background workers.
Simplebank is a financial services backend application built with Go that manages bank accounts and transfers. It utilizes a dual-protocol interface, providing both gRPC and REST APIs via Protocol Buffers to support different client communication requirements. The system implements a PostgreSQL data layer with versioned schema migrations and type-safe query generation. It handles financial operations through atomic fund transfers and balance change tracking to maintain consistent audit trails. The architecture includes an asynchronous task worker system using a message queue to offload long-
Offloads time-consuming background tasks to a separate worker system using a message queue.
MISP is an open-source threat intelligence sharing platform designed for collecting, storing, and distributing structured threat indicators and intelligence. At its core, it provides a distributed synchronization protocol for transferring events between instances, an attribute-based correlation engine that links matching indicators across events, and a REST API with an OpenAPI specification for programmatic access to threat data. The platform uses formal data formats for JSON, taxonomy, galaxy, and object templates to enable compatibility across tools and communities. The platform distinguish
Processes scheduled tasks and data operations asynchronously through a dedicated background worker queue system.
This is a distributed voting application designed to demonstrate a multi-service architecture. It uses stateless web frontends for submitting votes and viewing live results, with a Redis-backed queue to buffer incoming votes and a PostgreSQL database for persistent tallying. The application is built around asynchronous message queue processing, decoupling the vote submission from the tallying workflow. The project showcases how to deploy a multi-service application using container orchestration tools. It provides YAML-driven declarative deployment manifests for Docker Compose, Docker Swarm, a
Stores incoming votes in a Redis list for asynchronous consumption by a background worker.
protoactor-go is a framework for building concurrent and distributed systems in Go using the actor model. It provides a distributed actor system that enables isolated entities to communicate via asynchronous messaging and share state across a cluster. The framework implements a multi-language actor protocol, allowing interoperability between actors written in Go, C#, and Java. It further supports a virtual actor implementation, where actors are automatically instantiated across a network based on a unique identity. The system includes a supervision model for managing actor lifecycles and fau
Distributes incoming messages across pools of worker actors to balance load and increase throughput.
Flask-SocketIO is a Flask extension that integrates the Socket.IO real-time communication protocol into Flask applications, enabling persistent bidirectional connections between servers and clients. It transparently switches between WebSocket and HTTP long-polling while maintaining a persistent session identity, and provides a full event-driven framework for real-time web application communication. The library distinguishes itself with support for namespace multiplexing, allowing event handlers to be isolated into separate namespaces with independent lifecycle and routing. It also offers room
Uses a shared message queue to synchronize events across multiple worker processes for horizontal scaling.
Dramatiq is a distributed task queue and workload manager used to offload function execution to background workers. It functions as an asynchronous task orchestrator that enables the distribution of computational tasks across a cluster using a pluggable transport layer supporting RabbitMQ and Redis. The framework provides specialized tools for complex task orchestration, including the ability to link background jobs into sequences, pipelines, and barriers. It further manages distributed concurrency through the use of shared mutexes, rate limiters, and exponential backoff retries to prevent re
Provides background processing components that consume and execute tasks from declared message queues.
Ockam 是一个端到端加密框架和分布式身份提供商,旨在建立应用与设备之间的安全通信。它提供了一个安全网络覆盖层,利用加密身份和基于属性的访问控制来实现零信任网络访问。 该项目的特色在于元数据驱动的多跳路由和可插拔传输层,允许加密流量跨越不同的网络拓扑移动,而无需虚拟 IP 覆盖。它专门通过将原始 TCP 流量封装到加密通道中,为遗留应用实现安全隧道,从而通过出站中继实现私有网络连接和防火墙绕过。 该平台涵盖了广泛的功能,包括分布式身份管理、加密凭证颁发与验证,以及有状态并发 Actor 的执行。它还提供用于云规模节点配置和使用基础设施即代码模板进行自动化部署的工具。
Directs communication between decoupled actor components using a registry to resolve addresses and relay messages.
Ockam 是一个零信任网络框架,旨在通过基于身份的网络覆盖层来保护分布式应用之间的数据传输。它提供了建立双向认证和端到端加密连接所需的基础原语,从而消除了对传统网络层安全性的依赖。 该项目的特点是使用基于属性的访问控制和可验证凭证来大规模管理信任。它实现了加密身份轮换以保持身份连续性,并与硬件支持的密钥管理系统集成,从而将私钥安全地存储在安全隔离区或云密钥管理服务中。 该平台涵盖了广泛的功能,包括多跳二进制路由和基于中继的网络桥接,以连接不同的网络。它可以将传统的 TCP 或 Kafka 流量封装在安全隧道中,使私有服务能够在不暴露监听端口的情况下进行通信。此外,它还采用有状态的 Actor 模型,在分布式节点间异步处理消息。 部署方面,它支持通过基础设施即代码 (IaC) 模板在云环境中配置安全节点和网关。
Sends data to specific worker addresses through a router that resolves locations and manages delivery across different contexts.
useSend is an email marketing platform and transactional delivery system built to manage bulk campaigns and individual messages using AWS SES for delivery. It provides a RESTful API, an SMTP relay gateway, and a visual management interface for designing HTML templates and coordinating email broadcasts. The platform distinguishes itself by translating legacy SMTP traffic into modern API calls and implementing an event-driven webhook system. This system pushes real-time delivery, bounce, and click notifications to external servers using HMAC-SHA256 signed payloads to ensure authenticity. The s
Processes bulk email campaigns and scheduled messages using background workers for high throughput.
这是一个 PHP 兼容性库,为原生 ctype 扩展提供了 Polyfill。它作为一个可移植性层,实现了缺失的核心语言函数,以确保跨不同服务器环境和 PHP 版本的一致行为。 该库专注于字符类型验证,使用纯 PHP 代码重现原生扩展函数的行为。它使用运行时环境检测来识别原生扩展是否缺失,并将回退实现直接注入全局命名空间,以防止冲突并保持代码的可移植性。
Runs background processes that poll transports for pending messages and execute their associated handlers.
This project is a headless WhatsApp client and automation server that exposes multi-device functionality as a REST API gateway. It provides a centralized interface for managing multiple account sessions, automating messages, and developing bots through a standardized protocol. The system distinguishes itself by providing a secure webhook provider that forwards real-time messages and reactions to external URLs using signature verification. It includes specialized media utilities for converting images and GIFs into the WebP format required for stickers, as well as tools to simulate active user
Uses Go routines to process incoming events and outgoing messages without blocking the main execution thread.
The Cloudflare Workers Rust SDK is a toolchain and framework for building high-performance serverless functions using Rust. It compiles code into WebAssembly modules that execute within a sandboxed host environment at the edge. The project provides a typed remote procedure call system for invoking functions across different worker instances and utilizes procedural macros for mapping these distributed interfaces. It uses trait-based bindings and standardized wrappers to provide type-safe access to platform services and web-standard request and response objects. The SDK covers a broad range of
Implements background processing components that consume and execute tasks from message queues.
S3Scanner is a security tool for auditing public access and misconfigured permissions across S3-compatible storage providers. It functions as a storage auditor and security scanner designed to identify open buckets, enumerate publicly accessible objects, and exfiltrate data from misconfigured cloud environments. The project is distinguished by its integration with message brokers, allowing it to consume target lists for large-scale cloud infrastructure audits. It also provides utilities for dumping the entire contents of misconfigured buckets to local directories and calculating the total siz
Uses background workers to consume targets from message queues for distributed high-volume scanning.