29 repositorios
Libraries and connectors for interacting with message queuing and streaming systems.
Distinguishing note: Focuses on message broker connectivity rather than general database storage.
Explore 29 awesome GitHub repositories matching data & databases · Message Queue Integrations. Refine with filters or upvote what's useful.
FFmpeg is a cross-platform multimedia framework designed for the recording, conversion, and streaming of audio and video content. It functions as a comprehensive toolkit that provides both a command-line utility for direct media manipulation and a collection of low-level libraries for integration into custom applications. At its core, the project utilizes a packet-based stream engine and a format-agnostic abstraction layer to handle diverse media standards, containers, and network protocols. The framework distinguishes itself through a modular, graph-based filter execution model that allows f
Publishes or subscribes to media streams using message brokers to integrate live content into distributed messaging architectures.
This repository serves as a comprehensive collection of practical demonstrations and tutorials for building enterprise-ready Java applications using the Spring Boot framework. It provides structured guidance on core development topics, including the implementation of inversion-of-control containers, auto-configuration mechanisms, and convention-over-configuration patterns to simplify the assembly of complex systems. The project distinguishes itself by offering implementation patterns for diverse architectural requirements, such as microservices development, reactive programming models for hig
Implements asynchronous communication and service decoupling using message queuing systems.
OpenFaaS is a serverless function platform that provides a container-native framework for deploying and managing event-driven code. It functions as an abstraction layer over container orchestrators, allowing developers to package code into scalable functions that run across Kubernetes clusters or edge computing environments. The platform distinguishes itself through a developer-centric runtime that utilizes standardized language templates and automated build pipelines to simplify the creation of container images. It features a central API gateway that manages request routing, authentication,
Python¶ These are the official Python 3 templates maintained by OpenFaaS Ltd. The python3-http template is recommended for most Python functions. Use python3-http-debian when a dependency requires native compilati
Vector is a high-performance observability data pipeline designed to collect, transform, and route logs, metrics, and traces across distributed infrastructure. It functions as a modular engine that decouples data ingestion from processing and transmission, utilizing a component-based architecture to connect diverse sources to multiple destinations. The project distinguishes itself through a focus on reliability and flow control. It implements backpressure-aware data movement to prevent data loss during traffic spikes and utilizes disk-backed event buffering to ensure durability during network
Streams processed metric data into database instances using gRPC with support for authentication and compression.
SpringBoot-Labs is a collection of educational resources and reference implementations for Java backend architecture and distributed systems. It provides practical lab guides and code samples focused on building applications with the Spring Boot framework and designing scalable microservices architectures. The project specifically covers service governance and distributed cloud deployment patterns using Spring Cloud and Spring Cloud Alibaba. It includes a dedicated kit for microservices and a guide for executing remote procedure calls and managing service discovery via the Dubbo protocol. Th
Provides libraries and connectors for interacting with message queuing and streaming systems.
Yudao-cloud is a Java-based enterprise application platform designed for building scalable backend systems. It provides a modular architecture that supports both monolithic deployment for simplified development and microservices-based scaling for complex distributed environments. The platform functions as a comprehensive development framework that utilizes Spring Boot and distributed service orchestration patterns. It includes a centralized configuration manager to handle service discovery, dynamic settings, and reliable communication between independent components within a cloud-native envir
Integrates essential infrastructure like databases, caches, and message queues to support data storage and background processing requirements.
Telegraf is a modular, cross-platform telemetry pipeline designed to collect, process, and route metrics from diverse infrastructure, applications, and hardware. It functions as a server-side middleware that normalizes heterogeneous data into a unified format, enabling consistent monitoring across complex environments. By utilizing a plugin-driven architecture, the agent manages the entire lifecycle of telemetry data from initial ingestion to final transmission. The project distinguishes itself through a declarative, configuration-driven execution model that allows users to define complex dat
Reads metrics or message payloads from message brokers and distributed queue services using standard protocols.
Open-IM-Server is an instant messaging server and cross-platform messaging backend designed for real-time text and media communication. It provides a distributed chat architecture that coordinates user sessions and group communications across multiple server nodes to support millions of concurrent users. The system features a scalable message distribution model that utilizes asynchronous message pipelines and distributed queueing to maintain reliability during traffic spikes. It employs multi-level data caching across local memory and distributed stores to reduce latency and uses a document-b
Utilizes asynchronous message pipelines and distributed queueing to maintain reliability during traffic spikes.
VictoriaMetrics is a high-performance, scalable time series database and observability platform designed for long-term storage and analysis of metric, log, and trace data. It functions as a unified backend for monitoring ecosystems, offering full compatibility with industry-standard protocols and query languages. The system is built to handle massive data volumes through a distributed architecture that supports horizontal scaling and efficient data lifecycle management. The platform distinguishes itself through a storage engine that utilizes consistent hashing for data sharding and log-struct
Streams metric data to message queues using remote write protocols with support for compression and authentication.
Quarkus is a Kubernetes-native Java framework designed for building high-performance, memory-efficient applications. It utilizes ahead-of-time native compilation to transform Java code into standalone, optimized binaries that eliminate the need for a virtual machine, enabling rapid startup and reduced memory consumption. By performing code augmentation during the build phase, it shifts heavy processing tasks away from runtime, ensuring that applications are optimized for cloud-native environments. The framework distinguishes itself through a unified approach to reactive and imperative program
Maps messaging channels to specific broker exchanges or queues with options to declare new infrastructure or connect to existing resources.
Watermill is a Go library for building event-driven applications. It provides a framework for implementing event-driven architectures through a consistent interface for message publishing and subscribing, featuring a pub/sub message router and event stream processing pipelines. The library uses interface-based abstractions to decouple message producers and consumers from the underlying message broker. This allows different messaging technologies to be swapped via plugin-based broker integration. It employs a router-driven dispatch system and middleware-based pipelines to manage cross-cutting
Connects Go services to various message brokers using a consistent interface for publishing and subscribing.
Apache JMeter is a Java-based performance testing tool and multi-protocol traffic simulator used to analyze the stability and scalability of servers and networks. It functions as a distributed load testing framework that coordinates remote worker nodes from a single controller to generate high volumes of concurrent traffic. The project is distinguished by its ability to simulate traffic across diverse backend systems, including HTTP, JDBC, LDAP, JMS, FTP, and TCP. It provides a headless command-line interface for automated execution and a reporting system that transforms raw sample logs into
Interacts with JMS message queues using point-to-point or publisher-subscriber models.
Jeesite is a full-stack low-code development framework designed for building enterprise administrative portals using Spring Boot, MyBatis, and Vue. It functions as a comprehensive platform for creating administrative dashboards with integrated role-based access control and organizational data permission systems. The framework distinguishes itself through a combination of automated CRUD code generation and an integrated RAG platform that connects large language models to enterprise data via vector stores. It further incorporates a BPMN-based workflow engine to automate complex business process
Integrates distributed message brokers to process asynchronous tasks and event-driven messages.
goim
Queues outgoing messages through Kafka for non-blocking delivery, fan-out, and aggregated room delivery.
JavaTutorial is a specialized knowledge base and set of study guides focused on backend engineering, the Java ecosystem, distributed systems, and database internals. It serves as a technical reference for engineers, providing structured learning paths and curated content designed for Java backend developer interview preparation. The resource distinguishes itself through deep-dive analyses of internal mechanics, including JVM memory management, garbage collection algorithms, and the internal architecture of the Spring Framework. It provides detailed studies on database internals specifically f
Covers configuration of cluster operations and broker internals to implement reliable messaging queues.
Beanstalkd is a distributed work queue server designed for asynchronous job processing. It functions as a standalone server that distributes background tasks between producers and consumers to improve application responsiveness and throughput. The system organizes tasks using numeric priority levels to ensure critical work is processed first. It manages the job lifecycle through discrete states and uses a simple line-based text protocol over TCP for communication. To ensure reliability, the server persists job data to a sequential disk log, allowing the queue state to be recovered after a sy
Acts as a message broker to decouple system components via a durable task queue.
Materialize is a streaming SQL database that continuously ingests live data from sources such as Kafka, Redpanda, PostgreSQL, and MySQL, and incrementally maintains materialized views. It provides a PostgreSQL-compatible query engine that accepts standard SQL over the PostgreSQL wire protocol, enabling any existing SQL client or BI tool to query real-time data. The system also includes a Model Context Protocol (MCP) server that exposes live materialized view data to AI agents, providing fresh context without polling. Materialize distinguishes itself through its ability to offer configurable c
Configures starting offsets for Kafka topic consumption in streaming SQL sources.
kcat es un cliente de interfaz de línea de comandos para Apache Kafka utilizado para producir, consumir y depurar mensajes utilizando el protocolo nativo. Proporciona un conjunto de herramientas para interactuar con clusters de Kafka, incluyendo un depurador de protocolos para inspeccionar metadatos del cluster y un gestor de transacciones para manejar lotes de mensajes atómicos. El proyecto cuenta con un decodificador de esquemas Avro especializado que convierte mensajes codificados en binario en JSON legible por humanos mediante la integración con registros de esquemas remotos o archivos locales. Además, incluye un simulador en memoria que permite probar la lógica del productor y consumidor simulando el comportamiento efímero del broker sin requerir infraestructura externa. El conjunto de herramientas cubre una amplia gama de operaciones de mensajería, incluyendo soporte para grupos de consumidores balanceados, búsqueda de offsets basada en marcas de tiempo y streaming de datos transaccionales desde la entrada estándar. También proporciona utilidades para la configuración de seguridad de la conexión y la inspección de metadatos del cluster.
Serves as a standalone command-line utility for producing, consuming, and debugging messages within Kafka clusters.
Kafkacat es un conjunto de utilidades de línea de comandos para interactuar con clusters de Apache Kafka. Proporciona un binario no JVM para producir y consumir mensajes, inspeccionar metadatos del cluster y depurar el protocolo Kafka a través de la terminal. La herramienta funciona como un productor y consumidor capaz de enviar datos desde archivos o entrada estándar y leer mensajes de temas y particiones específicos. Incluye un inspector de metadatos para recuperar el estado del cluster y las configuraciones de partición en texto plano o JSON, así como un depurador de protocolos para inspeccionar offsets de mensajes, marcas de tiempo y cargas útiles binarias. El proyecto cubre la deserialización de datos utilizando esquemas o decodificadores primitivos y consultas basadas en offset para recuperar datos de marcas de tiempo precisas. También proporciona utilidades de simulación para ejecutar brokers efímeros en memoria para pruebas de integración y benchmarking de rendimiento.
Provides a high-performance producer and consumer for sending data to and reading messages from Kafka.
ServiceStack es un framework web de alto rendimiento para .NET diseñado para construir APIs con seguridad de tipos utilizando objetos de solicitud y respuesta fuertemente tipados. Funciona como un motor de API basado en mensajes que desacopla la lógica de negocio de la capa de transporte, permitiendo que los servicios se expongan a través de múltiples protocolos, incluyendo HTTP, gRPC y varios proveedores de colas de mensajes. El framework se distingue por su generador de API con seguridad de tipos, que produce SDKs de cliente nativos y objetos de transferencia de datos (DTOs) a partir de metadatos de servicio en múltiples lenguajes. También incluye un gateway de servicios distribuidos para la orquestación de microservicios, un ORM code-first para traducir objetos C# directamente a registros de base de datos y un sistema centralizado de gestión de identidad y acceso para un acceso seguro basado en tokens. Su superficie de capacidades más amplia cubre la mensajería asíncrona y el streaming de eventos en tiempo real mediante Pub-Sub y Server-Sent Events. Proporciona un soporte integral de serialización de datos para formatos como JSON, XML, ProtoBuf y MessagePack, junto con flujos de autenticación integrados que incluyen JWT, claves API y autenticación de paso a paso. Herramientas adicionales incluyen la generación automatizada de API CRUD, ejecución de trabajos en segundo plano y plantillas de dashboard administrativo.
Executes service logic across multiple transport layers including RabbitMQ, Redis, SQS, and Azure Service Bus.