29 个仓库
Libraries and connectors for interacting with message queuing and streaming systems.
Distinguishing note: Focuses on message broker connectivity rather than general database storage.
Explore 29 awesome GitHub repositories matching data & databases · Message Queue Integrations. Refine with filters or upvote what's useful.
FFmpeg is a cross-platform multimedia framework designed for the recording, conversion, and streaming of audio and video content. It functions as a comprehensive toolkit that provides both a command-line utility for direct media manipulation and a collection of low-level libraries for integration into custom applications. At its core, the project utilizes a packet-based stream engine and a format-agnostic abstraction layer to handle diverse media standards, containers, and network protocols. The framework distinguishes itself through a modular, graph-based filter execution model that allows f
Publishes or subscribes to media streams using message brokers to integrate live content into distributed messaging architectures.
This repository serves as a comprehensive collection of practical demonstrations and tutorials for building enterprise-ready Java applications using the Spring Boot framework. It provides structured guidance on core development topics, including the implementation of inversion-of-control containers, auto-configuration mechanisms, and convention-over-configuration patterns to simplify the assembly of complex systems. The project distinguishes itself by offering implementation patterns for diverse architectural requirements, such as microservices development, reactive programming models for hig
Implements asynchronous communication and service decoupling using message queuing systems.
OpenFaaS is a serverless function platform that provides a container-native framework for deploying and managing event-driven code. It functions as an abstraction layer over container orchestrators, allowing developers to package code into scalable functions that run across Kubernetes clusters or edge computing environments. The platform distinguishes itself through a developer-centric runtime that utilizes standardized language templates and automated build pipelines to simplify the creation of container images. It features a central API gateway that manages request routing, authentication,
Python¶ These are the official Python 3 templates maintained by OpenFaaS Ltd. The python3-http template is recommended for most Python functions. Use python3-http-debian when a dependency requires native compilati
Vector is a high-performance observability data pipeline designed to collect, transform, and route logs, metrics, and traces across distributed infrastructure. It functions as a modular engine that decouples data ingestion from processing and transmission, utilizing a component-based architecture to connect diverse sources to multiple destinations. The project distinguishes itself through a focus on reliability and flow control. It implements backpressure-aware data movement to prevent data loss during traffic spikes and utilizes disk-backed event buffering to ensure durability during network
Streams processed metric data into database instances using gRPC with support for authentication and compression.
SpringBoot-Labs is a collection of educational resources and reference implementations for Java backend architecture and distributed systems. It provides practical lab guides and code samples focused on building applications with the Spring Boot framework and designing scalable microservices architectures. The project specifically covers service governance and distributed cloud deployment patterns using Spring Cloud and Spring Cloud Alibaba. It includes a dedicated kit for microservices and a guide for executing remote procedure calls and managing service discovery via the Dubbo protocol. Th
Provides libraries and connectors for interacting with message queuing and streaming systems.
Yudao-cloud is a Java-based enterprise application platform designed for building scalable backend systems. It provides a modular architecture that supports both monolithic deployment for simplified development and microservices-based scaling for complex distributed environments. The platform functions as a comprehensive development framework that utilizes Spring Boot and distributed service orchestration patterns. It includes a centralized configuration manager to handle service discovery, dynamic settings, and reliable communication between independent components within a cloud-native envir
Integrates essential infrastructure like databases, caches, and message queues to support data storage and background processing requirements.
Telegraf is a modular, cross-platform telemetry pipeline designed to collect, process, and route metrics from diverse infrastructure, applications, and hardware. It functions as a server-side middleware that normalizes heterogeneous data into a unified format, enabling consistent monitoring across complex environments. By utilizing a plugin-driven architecture, the agent manages the entire lifecycle of telemetry data from initial ingestion to final transmission. The project distinguishes itself through a declarative, configuration-driven execution model that allows users to define complex dat
Reads metrics or message payloads from message brokers and distributed queue services using standard protocols.
Open-IM-Server is an instant messaging server and cross-platform messaging backend designed for real-time text and media communication. It provides a distributed chat architecture that coordinates user sessions and group communications across multiple server nodes to support millions of concurrent users. The system features a scalable message distribution model that utilizes asynchronous message pipelines and distributed queueing to maintain reliability during traffic spikes. It employs multi-level data caching across local memory and distributed stores to reduce latency and uses a document-b
Utilizes asynchronous message pipelines and distributed queueing to maintain reliability during traffic spikes.
VictoriaMetrics is a high-performance, scalable time series database and observability platform designed for long-term storage and analysis of metric, log, and trace data. It functions as a unified backend for monitoring ecosystems, offering full compatibility with industry-standard protocols and query languages. The system is built to handle massive data volumes through a distributed architecture that supports horizontal scaling and efficient data lifecycle management. The platform distinguishes itself through a storage engine that utilizes consistent hashing for data sharding and log-struct
Streams metric data to message queues using remote write protocols with support for compression and authentication.
Quarkus is a Kubernetes-native Java framework designed for building high-performance, memory-efficient applications. It utilizes ahead-of-time native compilation to transform Java code into standalone, optimized binaries that eliminate the need for a virtual machine, enabling rapid startup and reduced memory consumption. By performing code augmentation during the build phase, it shifts heavy processing tasks away from runtime, ensuring that applications are optimized for cloud-native environments. The framework distinguishes itself through a unified approach to reactive and imperative program
Maps messaging channels to specific broker exchanges or queues with options to declare new infrastructure or connect to existing resources.
Watermill is a Go library for building event-driven applications. It provides a framework for implementing event-driven architectures through a consistent interface for message publishing and subscribing, featuring a pub/sub message router and event stream processing pipelines. The library uses interface-based abstractions to decouple message producers and consumers from the underlying message broker. This allows different messaging technologies to be swapped via plugin-based broker integration. It employs a router-driven dispatch system and middleware-based pipelines to manage cross-cutting
Connects Go services to various message brokers using a consistent interface for publishing and subscribing.
Apache JMeter is a Java-based performance testing tool and multi-protocol traffic simulator used to analyze the stability and scalability of servers and networks. It functions as a distributed load testing framework that coordinates remote worker nodes from a single controller to generate high volumes of concurrent traffic. The project is distinguished by its ability to simulate traffic across diverse backend systems, including HTTP, JDBC, LDAP, JMS, FTP, and TCP. It provides a headless command-line interface for automated execution and a reporting system that transforms raw sample logs into
Interacts with JMS message queues using point-to-point or publisher-subscriber models.
Jeesite is a full-stack low-code development framework designed for building enterprise administrative portals using Spring Boot, MyBatis, and Vue. It functions as a comprehensive platform for creating administrative dashboards with integrated role-based access control and organizational data permission systems. The framework distinguishes itself through a combination of automated CRUD code generation and an integrated RAG platform that connects large language models to enterprise data via vector stores. It further incorporates a BPMN-based workflow engine to automate complex business process
Integrates distributed message brokers to process asynchronous tasks and event-driven messages.
goim
Queues outgoing messages through Kafka for non-blocking delivery, fan-out, and aggregated room delivery.
JavaTutorial is a specialized knowledge base and set of study guides focused on backend engineering, the Java ecosystem, distributed systems, and database internals. It serves as a technical reference for engineers, providing structured learning paths and curated content designed for Java backend developer interview preparation. The resource distinguishes itself through deep-dive analyses of internal mechanics, including JVM memory management, garbage collection algorithms, and the internal architecture of the Spring Framework. It provides detailed studies on database internals specifically f
Covers configuration of cluster operations and broker internals to implement reliable messaging queues.
Beanstalkd is a distributed work queue server designed for asynchronous job processing. It functions as a standalone server that distributes background tasks between producers and consumers to improve application responsiveness and throughput. The system organizes tasks using numeric priority levels to ensure critical work is processed first. It manages the job lifecycle through discrete states and uses a simple line-based text protocol over TCP for communication. To ensure reliability, the server persists job data to a sequential disk log, allowing the queue state to be recovered after a sy
Acts as a message broker to decouple system components via a durable task queue.
Materialize is a streaming SQL database that continuously ingests live data from sources such as Kafka, Redpanda, PostgreSQL, and MySQL, and incrementally maintains materialized views. It provides a PostgreSQL-compatible query engine that accepts standard SQL over the PostgreSQL wire protocol, enabling any existing SQL client or BI tool to query real-time data. The system also includes a Model Context Protocol (MCP) server that exposes live materialized view data to AI agents, providing fresh context without polling. Materialize distinguishes itself through its ability to offer configurable c
Configures starting offsets for Kafka topic consumption in streaming SQL sources.
kcat 是一个用于 Apache Kafka 的命令行接口客户端,用于使用原生线路协议生产、消费和调试消息。它提供了一套用于与 Kafka 集群交互的工具,包括用于检查集群元数据的协议调试器和用于处理原子消息批次的事务管理器。 该项目具有一个专门的 Avro 模式解码器,通过与远程模式注册表或本地文件集成,将二进制编码的消息转换为人类可读的 JSON。此外,它还包含一个内存模拟器,允许通过模拟短暂的 Broker 行为来测试生产者和消费者逻辑,而无需外部基础设施。 该工具集涵盖了广泛的消息传递操作,包括平衡消费者组支持、基于时间戳的偏移量查找以及来自标准输入的事务性数据流。它还提供了用于连接安全配置和集群元数据检查的实用程序。
Serves as a standalone command-line utility for producing, consuming, and debugging messages within Kafka clusters.
Kafkacat 是一套用于与 Apache Kafka 集群交互的命令行实用程序。它提供了一个非 JVM 二进制文件,用于生产和消费消息、检查集群元数据以及通过终端调试 Kafka 协议。 该工具的功能包括作为生产者和消费者,能够从文件或标准输入推送数据,并从特定主题和分区读取消息。它包括一个元数据检查器,以纯文本或 JSON 格式检索集群状态和分区配置,以及一个用于检查消息偏移量、时间戳和二进制负载的协议调试器。 该项目涵盖使用模式或原始解码器进行数据反序列化,以及基于偏移量的查询以从精确时间戳检索数据。它还提供模拟实用程序来运行短暂的内存中 Broker,用于集成测试和性能基准测试。
Provides a high-performance producer and consumer for sending data to and reading messages from Kafka.
ServiceStack 是一个高性能 .NET Web 框架,专为使用强类型请求和响应对象构建类型安全的 API 而设计。它作为一个基于消息的 API 引擎,将业务逻辑与传输层解耦,允许通过包括 HTTP、gRPC 和各种消息队列提供商在内的多种协议公开服务。 该框架的特点是其类型安全的 API 生成器,它从跨多种语言的服务元数据中生成原生客户端 SDK 和数据传输对象。它还包括用于微服务编排的分布式服务网关、将 C# 对象直接转换为数据库记录的代码优先 ORM,以及用于安全令牌访问的集中式身份和访问管理系统。 其更广泛的功能面涵盖通过 Pub-Sub 和服务器发送事件(SSE)进行的异步消息传递和实时事件流。它为 JSON、XML、ProtoBuf 和 MessagePack 等格式提供全面的数据序列化支持,以及包括 JWT、API 密钥和阶梯式认证在内的集成认证流程。其他工具包括自动 CRUD API 生成、后台作业执行和管理仪表盘模板。
Executes service logic across multiple transport layers including RabbitMQ, Redis, SQS, and Azure Service Bus.