258 repositorios
Platforms for processing and delivering data streams in real-time.
Distinguishing note: Focuses on event-driven streaming integrated into the database layer.
Explore 258 awesome GitHub repositories matching data & databases · Real-Time Data Streaming. Refine with filters or upvote what's useful.
Este proyecto es un directorio curado por la comunidad de endpoints de servicios REST y GraphQL diseñado para ayudar a los desarrolladores a descubrir e integrar fuentes de datos de terceros. Funciona como un registro centralizado donde los servicios externos se organizan por dominio para facilitar la creación rápida de prototipos y el desarrollo de aplicaciones. El registro se basa en un modelo de contribución revisado por pares, utilizando control de versiones distribuido para gestionar las actualizaciones y garantizar la precisión de los endpoints listados. Para mantener una alta calidad de los datos, el proyecto emplea validación basada en esquemas para todos los envíos entrantes y compila los datos estructurados en un sitio web estático y buscable para una recuperación eficiente. El directorio cubre un amplio espectro de capacidades de integración, incluyendo recuperación de datos financieros, servicios de geolocalización y varias API de utilidad para tareas como detección de idiomas, procesamiento de medios y verificación de identidad. Al proporcionar un índice centralizado de estos servicios, el proyecto ayuda a los desarrolladores a identificar proveedores de datos confiables para diversos requisitos funcionales.
Provides structured interfaces for retrieving global weather information.
Este proyecto es un directorio curado por la comunidad de software de código abierto diseñado para su implementación en entornos de servidores privados y laboratorios domésticos. Sirve como un recurso integral para descubrir alternativas independientes y autohospedadas a los servicios en la nube convencionales, permitiendo a los usuarios mantener la propiedad total de los datos y el control sobre su infraestructura digital. El directorio está estructurado a través de una taxonomía jerárquica que organiza una vasta colección de aplicaciones en categorías lógicas, que van desde la gestión de medios y análisis de datos hasta la comunicación privada y herramientas de productividad en equipo. Se distingue por un proceso de revisión por pares colaborativo, donde los miembros de la comunidad validan la calidad y relevancia de cada envío para garantizar que el directorio siga siendo preciso y confiable. El proyecto cubre una amplia superficie de capacidades, incluyendo automatización de infraestructura, implementación de servicios basados en contenedores y gestión de configuración declarativa. Estas herramientas ayudan a los usuarios a mantener entornos de servidor reproducibles y gestionar dependencias de servicios complejas en hardware privado. El directorio se mantiene como un repositorio con control de versiones, asegurando que todas las actualizaciones y cambios impulsados por la comunidad sean rastreados y transparentes.
Routes and publishes live video or audio feeds across multiple protocols to enable low-latency distribution and recording.
OpenCV is an open-source computer vision library and visual analysis toolkit. It provides a framework for processing static images and dynamic video frames to analyze visual data and extract information using deep learning. The project functions as a real-time image processing framework, enabling the execution of vision algorithms on live video streams for immediate analysis and data processing. The toolkit covers a broad range of capabilities including image pattern recognition, real-time video analysis, and visual data extraction. It also supports automated visual inspection for detecting
Executes vision algorithms on live video streams for immediate visual analysis and data processing.
Dubbo is a Java RPC framework and microservices governance platform designed for high-performance remote procedure calls in distributed architectures. It provides the foundational components necessary to connect distributed services across a network, including a binary data serialization library and a distributed service registry. The platform distinguishes itself through a comprehensive governance suite that manages service discovery, load balancing, and traffic routing. It enables precise control over network traffic via conditional routing and a pluggable extension mechanism based on a ser
Provides high-performance data streaming capabilities between clients and servers using the gRPC protocol.
This repository serves as a comprehensive research platform and toolkit for advancing machine learning, quantum computing, and large-scale scientific data analysis. It provides foundational frameworks for developing complex algorithmic systems, offering the necessary infrastructure for distributed training, computational graph execution, and high-performance model development. The project distinguishes itself by integrating specialized research domains with robust, privacy-preserving methodologies. It supports diverse scientific discovery through tools for quantum simulation, physics-informed
Combines diverse meteorological inputs from satellite and gauge-based sources into a unified framework to improve predictive accuracy.
SurrealDB is a multi-model database engine designed to store and query document, graph, relational, and vector data within a single ACID-compliant platform. It functions as an AI-native data store, integrating vector search, graph traversal, and machine learning model execution directly into its query layer. By providing a unified declarative query language, the platform eliminates the need for external middleware to synchronize data across different storage models. The platform distinguishes itself through its ability to manage agent memory and complex workflows natively. It allows developer
Powers reactive, real-time experiences using built-in subscriptions, event triggers, and streaming updates.
graphql-engine is an automated GraphQL API engine that transforms database tables and relationships into a queryable GraphQL schema. It functions as a federation gateway and mapper, instantly generating APIs with built-in filtering, pagination, and mutations from existing databases and remote schemas. The project distinguishes itself through a fine-grained access control layer that enforces row-level and field-level permissions. It further provides a real-time data subscription server that converts standard queries into live streams and a system for triggering event-driven webhooks and notifi
Converts standard GraphQL queries into live queries that stream real-time updates to clients.
SRS is a real-time media server designed to ingest, route, and distribute live audio and video streams across various transport protocols. It functions as a multi-protocol stream relay, including a dedicated RTMP media gateway and a WebRTC signaling server to coordinate peer-to-peer media exchanges. The system features a multi-protocol relay engine that transforms incoming media packets between different transport formats without re-encoding. This allows it to serve as a video delivery proxy that routes live media from a single source to multiple concurrent viewers using diverse delivery prot
Provides a multi-protocol relay engine to synchronize and forward live media streams between diverse transport protocols.
Hyperframes is an HTML-to-video rendering engine and composition tool that transforms web layouts and CSS into encoded video files. It functions as a headless browser video pipeline and a distributed video rendering framework, allowing users to create seekable animations and programmatic motion designs using HTML, CSS, and JavaScript. The project differentiates itself as an AI agent video orchestrator, enabling the automation of video scripts and compositions through natural language prompts. It supports distributed video encoding by splitting rendering tasks across multiple serverless functi
Pipes captured browser frames directly to the encoder in real time to avoid writing intermediate files to disk.
Seata is a distributed transaction coordinator designed to ensure data consistency and atomicity across microservices. It provides a centralized framework for managing global transactions, preventing partial data updates across different databases and services. The project implements multiple transaction modes to balance consistency and performance. This includes an automatic mode that uses rollback logs to coordinate compensation without modifying business logic, a try-confirm-cancel pattern for resources lacking native ACID support, and a saga orchestration engine for managing long-lived bu
Uses gRPC for high-performance bidirectional data streaming between clients and coordinators.
Kotaemon is an orchestration framework designed for building modular, agentic workflows that integrate document processing, retrieval-augmented generation, and multi-step reasoning. It provides a comprehensive platform for developing document-based question answering systems, allowing users to chain language models, prompt templates, and external tools into complex, automated pipelines. The system distinguishes itself through a highly modular architecture that emphasizes component-based composition and schema-driven data exchange. It supports autonomous agents capable of decomposing complex q
Streams incremental results from reasoning pipelines to the user interface in real-time.
RxSwift is a reactive programming library for Swift that provides a framework for managing push-based data flows and composing asynchronous, event-based programs. It utilizes observable sequences and functional operators to transform and filter asynchronous sequences through a declarative approach. The library is distinguished by its ability to link asynchronous data streams directly to user interface elements, automating view updates via reactive data binding. It includes specialized tools for tracking UI control properties and events on the main thread, as well as the ability to encapsulate
Distributes a single subscription across multiple observers with support for replaying historical elements.
This project is a reactive, offline-first NoSQL database engine designed for JavaScript applications. It provides a robust framework for managing application state by synchronizing data across browsers, mobile devices, and server-side runtimes. By treating local storage as the primary source of truth, it enables applications to remain functional without network connectivity, automatically reconciling changes with remote backends once a connection is restored. The database distinguishes itself through a modular architecture that supports cross-environment synchronization and high-performance d
Provides observable queries and documents that push state changes to the application for dynamic, responsive user interfaces.
This project serves as a comprehensive technical reference for the architecture and design of data-intensive applications. It provides a structured analysis of the fundamental principles required to build reliable, scalable, and maintainable software systems, covering the core trade-offs inherent in modern data infrastructure. The repository explores the mechanics of distributed data management, including strategies for replication, partitioning, and achieving consensus across multiple nodes. It details the design of storage engines, indexing techniques, and transaction management models, whi
Matches incoming documents against stored queries in real time to notify users when data meets specific criteria.
RocketMQ is a distributed messaging and streaming platform designed for building event-driven applications. It serves as middleware to decouple services using publish-subscribe and request-reply patterns, and functions as a transactional messaging system that ensures atomicity by linking message delivery to local transaction outcomes. The platform includes specialized capabilities as a Kubernetes-native message broker for container orchestration environments and an MQTT broker for ingesting event data from mobile applications and hardware terminals. The system covers high-throughput data str
Provides a platform for processing and transforming continuous event streams in real-time to synchronize state and derive insights.
Deepface is a comprehensive deep learning library for facial recognition and demographic analysis. It provides a modular pipeline that handles the entire lifecycle of facial processing, including detection, geometric alignment, and the transformation of facial images into high-dimensional numerical vector embeddings for identity verification and similarity comparison. The library distinguishes itself through a model ensemble approach, which combines predictions from multiple pre-trained neural networks to improve classification accuracy and reduce bias. It also integrates advanced security fe
Processes live video feeds to perform continuous face recognition and attribute analysis.
This project is a feature-rich Go client library designed for interacting with Redis. It serves as a comprehensive interface for managing remote data stores, enabling developers to execute standard database commands, handle complex data structures, and perform asynchronous operations within Go applications. The library distinguishes itself through its support for advanced Redis capabilities, including connection pooling, pipelining, and transactional integrity. It provides specialized primitives for managing distributed clusters, including automated topology updates and request routing to sha
Supports append-only log structures for real-time event streaming and asynchronous data processing.
Vector is a high-performance observability data pipeline designed to collect, transform, and route logs, metrics, and traces across distributed infrastructure. It functions as a modular engine that decouples data ingestion from processing and transmission, utilizing a component-based architecture to connect diverse sources to multiple destinations. The project distinguishes itself through a focus on reliability and flow control. It implements backpressure-aware data movement to prevent data loss during traffic spikes and utilizes disk-backed event buffering to ensure durability during network
Routes processed logs and metrics into message topics for real-time streaming.
This project is a comprehensive framework for building AI-powered applications, providing a unified toolkit for orchestrating language models, autonomous agents, and interactive user interfaces. It serves as a central library for managing the entire lifecycle of AI interactions, from initial prompt generation and model provider abstraction to complex, multi-step reasoning and tool execution. The framework distinguishes itself through its deep integration with frontend development, specifically by enabling generative user interfaces that render dynamic components directly from model outputs. I
Transmits raw data and text tokens in real time to provide immediate feedback during generation.
This project is a comprehensive educational guide and framework for building web scrapers using Python. It provides a course-based approach to data extraction, combining a Python crawler framework with tutorials on web reverse engineering and network traffic analysis. The project distinguishes itself by covering advanced extraction challenges, including the decryption of obfuscated JavaScript and the bypass of anti-scraping measures. It specifically addresses mobile application scraping through the simulation of user interactions and the interception of network traffic. The capability surfac
Supports capturing rapidly changing information via persistent socket connections for live data streaming.