24 repositorios
Utilities designed to collect, merge, and unify data from multiple disparate sources or endpoints.
Explore 24 awesome GitHub repositories matching data & databases · Data Aggregation Tools. Refine with filters or upvote what's useful.
This project is a community-maintained, open-source repository that functions as a centralized directory for streaming metadata. It aggregates publicly available network stream links and organizes them into standardized, machine-readable playlist formats. By acting strictly as a metadata-only index, the platform enables users to access and organize live broadcast content across various third-party media playback applications without hosting or distributing any actual video files. The repository distinguishes itself through a collaborative, crowdsourced workflow where contributors actively mai
Merges distributed community updates into a unified, structured dataset of verified streaming links.
This project functions as a curated software directory and developer resource index, providing a centralized platform for discovering and evaluating high-quality open-source repositories. It serves as an aggregator that monitors trending software and educational resources, organizing them by technical domain and programming language to assist developers in identifying tools for their specific technical challenges. The directory distinguishes itself through a community-driven curation workflow, where repository lists are validated and updated based on collective developer consensus. This infor
Retrieves real-time repository metadata and contributor statistics directly from remote service endpoints.
This project is a serverless service that generates dynamic, themeable visual summaries of software development activity. It functions as an automated metadata visualizer, transforming raw platform logs and repository metrics into resolution-independent vector graphics that can be embedded directly into markdown environments. The service distinguishes itself by offering highly configurable, query-parameter-driven rendering that allows users to customize the visual presentation of their coding patterns, language proficiency, and repository details. It supports both real-time generation via ser
Merges disparate data points from multiple remote endpoints into a unified schema before rendering the final visual output.
graphql-engine is an automated GraphQL API engine that transforms database tables and relationships into a queryable GraphQL schema. It functions as a federation gateway and mapper, instantly generating APIs with built-in filtering, pagination, and mutations from existing databases and remote schemas. The project distinguishes itself through a fine-grained access control layer that enforces row-level and field-level permissions. It further provides a real-time data subscription server that converts standard queries into live streams and a system for triggering event-driven webhooks and notifi
Aggregates data from disparate remote endpoints into a unified schema to integrate custom business logic.
Vector is a high-performance observability data pipeline designed to collect, transform, and route logs, metrics, and traces across distributed infrastructure. It functions as a modular engine that decouples data ingestion from processing and transmission, utilizing a component-based architecture to connect diverse sources to multiple destinations. The project distinguishes itself through a focus on reliability and flow control. It implements backpressure-aware data movement to prevent data loss during traffic spikes and utilizes disk-backed event buffering to ensure durability during network
Centralizes data from multiple agents to scrub sensitive information, reformat logs, and sample streams before forwarding.
FossFLOW is an open source metadata search engine and data platform designed to aggregate and normalize repository information from multiple code hosting services. It functions as a developer productivity utility, enabling users to discover software projects and analyze contributor networks through a unified, searchable index. The platform distinguishes itself by utilizing vector-based semantic search, which converts project descriptions and code metadata into numerical embeddings to facilitate discovery based on conceptual relevance. To maintain a consistent view of disparate data, the syste
Aggregates and filters project metadata directly from remote code hosting service endpoints.
This project is a collection of educational resources and reference implementations for the Apache Flink stream processing framework. It provides a learning resource focused on mastering distributed stream processing through implementation guides, performance tuning tutorials, and practical examples. The repository features detailed walkthroughs for building real-time data pipelines using the DataStream and Table APIs. It includes specific integration examples for connecting Apache Flink with Kafka brokers and Elasticsearch indices, as well as reference implementations for real-time deduplica
Groups continuous data flows into temporal or count-based windows to perform periodic aggregations.
Jackett is a self-hosted background service that functions as a BitTorrent tracker aggregator and proxy. It enables automated media management applications to query multiple torrent indexers simultaneously by translating standardized search requests into site-specific formats and consolidating the resulting data into a single, unified feed. The service distinguishes itself through an adapter-based architecture that handles the complexities of disparate tracker interfaces and security protocols. It integrates with external proxy services to bypass anti-bot challenges and maintain persistent ac
Fetches and merges disparate data points from multiple remote endpoints into a unified schema.
This project is a static analysis engine and type checker designed for PHP codebases. It evaluates source code structure and type annotations to identify potential bugs, type mismatches, and logic errors without executing the application. By parsing code into an abstract syntax tree and applying a rule-based validation framework, it enforces code quality and safety standards across a project. What distinguishes this tool is its sophisticated type inference engine, which models dynamic language features, magic methods, and conditional types to maintain accuracy even in unconventional code. It
Collects and processes information across multiple files to support complex analysis rules that require global context.
Devhub is a cross-platform developer tool and event aggregator designed to monitor GitHub activities. It provides a unified interface for tracking issues, notifications, and user actions across multiple repositories, consolidating these updates into a single view to reduce notification clutter. The application utilizes a multi-column dashboard for organizing data streams via customizable filters and saved searches. This interface allows for the management of review queues, the monitoring of specific user actions, and the display of notification context without requiring navigation to the sour
Collects repository updates and user actions into a single view to reduce notification clutter.
Star History is a suite of utilities for visualizing the growth of GitHub repositories over time. It functions as a star growth visualizer, a repository comparison tool, a metric embedder for external websites, and a trending analytics dashboard. The project enables the analysis of star acquisition rates for multiple repositories on a single chart to determine relative growth. It also provides the ability to rank repositories by growth windows to identify rising projects. The system covers project analytics and open source benchmarking by generating time-series charts and growth reports. It
Aggregates star event data by polling the GitHub API for historical repository metrics.
RisingWave is a cloud-native streaming database and real-time analytics engine that uses standard SQL to process continuous data streams. It functions as a streaming data lakehouse, combining the capabilities of a streaming SQL database with a platform that integrates streaming ingestion with open table formats. The system is distinguished by its use of the PostgreSQL wire protocol, allowing it to integrate with existing SQL tools and drivers. It employs a decoupled compute and storage architecture, persisting streaming state and materialized views in cloud object storage to enable independen
Groups continuous event flows into time intervals to calculate periodic metrics.
Istanbul is a JavaScript code coverage tool and instrumentation engine that measures the execution of statements, lines, functions, and branches. It functions as a test coverage analysis tool capable of monitoring code across unit, functional, and browser tests to identify untested areas of a codebase. The project distinguishes itself through a transparent instrumentation engine that uses module loader hooks to inject tracking code without requiring manual source modifications. It supports distributed test reporting by aggregating fragmented coverage data from multiple concurrent processes in
Combines fragmented coverage reports from multiple concurrent execution processes into a single unified dataset.
This project is a Go shell scripting library and framework designed for writing automation scripts and CLI tools. It provides a concurrent data pipeline system for chaining sources, filters, and sinks to process text and JSON streams. The library distinguishes itself through a comprehensive toolkit for shell-like operations, including a text processing engine for regular expression filtering and frequency analysis, a filesystem utility toolkit for recursive search and path manipulation, and an integrated HTTP client wrapper for building data pipelines that fetch web content. The capability s
Includes tools for counting input lines and calculating unique entry frequencies to summarize data patterns.
GhostTrack is an open-source intelligence (OSINT) framework that aggregates geographic, network, and social identity information from public data sources. It functions as a digital footprint analyzer, collecting various pieces of publicly available information to build comprehensive profiles of target individuals. The framework combines multiple investigative capabilities into a single tool, including IP address geolocation, phone number intelligence, and social media username discovery. It distributes queries across external data services to maximize coverage and accuracy, resolving IP addre
Aggregates data from multiple external APIs by making asynchronous requests and normalizing responses.
Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis
Enables the construction of streaming data pipelines that perform real-time joins, sorts, and aggregations.
Pinot is a distributed, columnar analytical database designed for high-concurrency, low-latency query processing. It functions as a real-time OLAP datastore, enabling interactive, user-facing analytics by ingesting and querying massive datasets from both streaming and batch sources. The system architecture relies on a centralized controller for cluster coordination and a distributed segment-based storage model to ensure horizontal scalability. The platform distinguishes itself through a hybrid ingestion pipeline that unifies real-time event streams and historical batch data into a single quer
Consumes events from streaming sources to create a unified, queryable SQL view across microservice architectures.
SimpleCov es una herramienta de cobertura de código para Ruby y motor de análisis utilizado para rastrear qué líneas, ramas y métodos de código se ejecutan durante las pruebas. Funciona como un ejecutor de umbrales de cobertura y agregador de suites de pruebas, registrando datos de ejecución para identificar áreas no probadas de una aplicación. La herramienta se distingue por la capacidad de fusionar resultados de cobertura de procesos trabajadores paralelos y subprocesos en un único informe unificado. Soporta la comparación de líneas base para detectar regresiones de cobertura y puede recopilar datos de código ejecutado mediante métodos de evaluación dinámica, como los utilizados en motores de plantillas. Sus capacidades más amplias incluyen la generación de informes en múltiples formatos, agrupación de archivos fuente y filtrado de archivos mediante expresiones regulares. El sistema también proporciona una interfaz de línea de comandos para mostrar estadísticas y listar archivos no cubiertos.
Merges fragmented coverage data from parallel worker processes and multiple test suites into a single report.
Cube is a time-series analytics platform and event data store designed for real-time performance monitoring. It functions as a metrics engine that ingests timestamped event streams and persists raw logs to enable the computation of statistical summaries, quantiles, and histograms. The system distinguishes itself through a reactive processing model that automatically invalidates metric caches when new events arrive, ensuring query results remain current. It supports both real-time event streaming via persistent connections and the calculation of post hoc statistics from stored event sets. The
Converts raw event streams into aggregate statistics, quantiles, and histograms for high-level system observation.
This project is an AI model quota monitoring and management tool for development environments. It provides a centralized interface for tracking remaining request limits, managing multiple authorized API accounts, and orchestrating AI service access. The system synchronizes quota reset cycles and automates model activation through scheduled tasks. It enables the inspection of large language model capabilities, such as context window sizes and supported input types, while allowing models to be clustered into groups based on shared service categories. Real-time usage data is delivered through a
Implements a system to fetch and merge quota data from multiple remote endpoints into a unified view.