24 Repos
Techniques to collapse multiple concurrent network requests for the same resource.
Distinguishing note: Focuses on request collapsing, not general caching.
Explore 24 awesome GitHub repositories matching web development · Request Deduplication. Refine with filters or upvote what's useful.
SWR is a data fetching library that provides a collection of hooks for managing remote data synchronization, caching, and state updates in web applications. It employs a declarative approach to handle complex network request lifecycles and dependency chains, ensuring that client-side application state remains consistent with server data through automatic revalidation and background updates. The library distinguishes itself through a reactive cache layer that automatically synchronizes local state with remote sources based on component lifecycle events. It features event-driven revalidation, w
Collapses multiple concurrent requests for the same resource into a single network call using unique keys.
Vector is a high-performance observability data pipeline designed to collect, transform, and route logs, metrics, and traces across distributed infrastructure. It functions as a modular engine that decouples data ingestion from processing and transmission, utilizing a component-based architecture to connect diverse sources to multiple destinations. The project distinguishes itself through a focus on reliability and flow control. It implements backpressure-aware data movement to prevent data loss during traffic spikes and utilizes disk-backed event buffering to ensure durability during network
Eliminates redundant logs or metrics by comparing incoming data against a configurable cache of recent events.
Cube is a semantic data layer that provides a unified framework for defining business metrics, dimensions, and relationships across diverse data sources. By acting as a headless business intelligence engine, it transforms raw data into a governed model that can be queried via SQL, REST, and GraphQL interfaces. This architecture ensures consistent data definitions and logic across all downstream analytical applications and reporting tools. The platform distinguishes itself through its integrated conversational AI capabilities, which allow users to explore data using natural language. It orches
Protects data sources by deduplicating redundant analytical queries through managed queues.
VictoriaMetrics is a high-performance, scalable time series database and observability platform designed for long-term storage and analysis of metric, log, and trace data. It functions as a unified backend for monitoring ecosystems, offering full compatibility with industry-standard protocols and query languages. The system is built to handle massive data volumes through a distributed architecture that supports horizontal scaling and efficient data lifecycle management. The platform distinguishes itself through a storage engine that utilizes consistent hashing for data sharding and log-struct
Collapses redundant data points retrieved from multiple storage replicas into single samples to ensure accurate query results.
Thanos is a CNCF cloud native monitoring tool that provides a highly available and scalable extension to the Prometheus ecosystem. It functions as a global query engine, a long-term storage system, and a metric downsampler. The project enables a unified interface to aggregate and query metrics across multiple distributed clusters from a single view. It maintains historical data beyond local retention limits by persisting time-series metrics in object storage and eliminates data gaps by merging metrics from redundant server pairs. The system includes capabilities for reducing the resolution o
Implements metric sample deduplication to collapse redundant data points from multiple storage replicas for storage accuracy.
Groupcache is a distributed caching library designed to coordinate data retrieval and storage across a cluster of nodes. It functions as a peer-to-peer data store that uses consistent hashing to assign specific keys to canonical owners, ensuring that cached items remain predictable and accessible throughout the network. The system distinguishes itself through a request coalescing engine that merges concurrent requests for the same missing key into a single upstream fetch. This mechanism prevents redundant backend load by ensuring that only one process retrieves the required data while sharing
Prevents redundant backend fetches by merging concurrent requests for the same data into a single operation.
Coil is an image loading and caching pipeline designed for Android and Compose Multiplatform applications. It functions as a comprehensive loader, caching engine, and rendering utility that asynchronously fetches and displays images from network URLs, local storage, and multiplatform resource systems. The library distinguishes itself through a flexible fetcher-decoder pipeline and an interface-driven component registry, allowing for the integration of custom networking clients and decoders. It provides specialized support for rendering scalable vector graphics, animated formats such as GIF an
Implements request collapsing to combine multiple simultaneous requests for the same image resource into a single execution.
Falcor is a JavaScript library that models remote data as a single virtual JSON graph, providing a path-based query engine for efficient client-side data retrieval and updates. It represents multiple remote data sources as a unified document where entities are accessed via globally unique identity paths. The system distinguishes itself by treating the remote data model as a virtual JSON resource, allowing the client to query specific paths without managing individual endpoints. It uses a reference-aware graph model to handle many-to-many relationships and prevents data duplication. Network ef
Deduplicates requests during virtual graph traversal to reduce redundant data fetches.
The Prometheus Operator is a Kubernetes monitoring orchestrator and controller that manages Prometheus clusters and observability components through declarative custom resources. It functions as a custom resource controller that translates high-level Kubernetes resource definitions into the configuration files required by the underlying monitoring software. The project automates the deployment, scaling, and lifecycle of an observability stack, including the integration of components like Thanos and Alertmanager. It distinguishes itself by syncing monitoring targets, alerting rules, and scrape
Implements external labels and relabeling rules to prevent duplicate data samples in high-availability backends.
Soar is a suite of specialized tools designed for analyzing MySQL performance, advising on indexing, and optimizing SQL syntax. It functions as a performance analyzer, index advisor, and query optimizer to identify bottlenecks and suggest structural improvements for faster execution. The project distinguishes itself through a system for rewriting SQL statements into optimized equivalent versions using custom heuristic rules and patterns. It also features a dedicated index advisor that evaluates query patterns and database metadata to recommend the creation of new indexes. Its broader capabil
Provides query fingerprinting by replacing literals with placeholders for deduplication and grouping.
Nuke is an image loading system designed to fetch, cache, and render images and short video clips within a user interface. It functions as an asset caching engine and a resumable download manager to handle the retrieval of remote media files. The framework includes a request priority manager to coordinate duplicate requests and order loading based on visual importance. It employs an image prefetching strategy to load visuals into memory before they are required and uses an LRU asset cache to reduce network traffic. The system covers broader capabilities in traffic management and data storage
Collapses multiple concurrent network requests for the same image resource into a single shared promise.
Pholcus is a distributed web crawling system designed for large-scale data scraping. It employs a master-worker distribution model to coordinate high-concurrency scraping tasks across a network of remote client nodes, enabling both horizontal and vertical data collection. The system features a hot-loadable rule engine that allows extraction and navigation logic to be updated at runtime without restarting the process. It handles dynamic content through headless browser integration and bypasses bot detection using proxy rotation, automated user authentication, and simulated human behavior. The
Includes a request deduplication pipeline to prevent redundant network calls and infinite crawling loops.
HotChocolate is a GraphQL server implementation for .NET that handles queries, mutations, and subscriptions over HTTP. It provides code-first schema generation directly from C# classes and attributes, keeping the schema and business logic in sync without manual type definitions. The project also includes a GraphQL gateway that composes multiple GraphQL and REST endpoints into a single unified schema, along with a GraphQL IDE for exploring, testing, and debugging GraphQL APIs interactively. The project distinguishes itself through its comprehensive tooling ecosystem, which includes a GraphQL c
Merges concurrent requests for the same key so the data source is queried only once per batch.
re-frame ist ein funktionales Framework für den Bau von Single-Page-Anwendungen in ClojureScript. Es bietet eine zentralisierte, unveränderliche Datenbank, die als Single Source of Truth für den gesamten Anwendungszustand dient und einen strikten, unidirektionalen Datenfluss erzwingt, bei dem Events Zustandsübergänge und nachfolgende View-Updates auslösen. Das Framework zeichnet sich durch einen reaktiven Signal-Graphen und eine Interceptor-basierte Middleware-Pipeline aus. Indem Anwendungslogik als Sequenz datengetriebener Events und deklarativer Side-Effects behandelt wird, entkoppelt es die Geschäftslogik von der View-Schicht. Diese Architektur ermöglicht es Entwicklern, komplexe Zustandsübergänge und externe Operationen durch reine Funktionen zu verwalten, wobei sichergestellt wird, dass Side-Effects von einem separaten Interpreter statt durch imperative Aufrufe ausgeführt werden. Das System umfasst eine umfassende Suite an Funktionen für das Management der Anwendungsarchitektur, einschließlich reaktiver Datenableitung, abonnementbasierter View-Reconciliation und eventgesteuertem Zustandsmanagement. Es unterstützt fortgeschrittene Entwicklungs-Workflows wie Event-Tracing, State-Checkpointing und die Möglichkeit, Side-Effects für isolierte Tests zu stubben. Das Projekt ist für die Integration mit React konzipiert und nutzt Virtual-DOM-Reconciliation, um Benutzeroberflächen effizient zu aktualisieren. Es bietet ein robustes Set an Utilities für den Umgang mit Cross-Cutting Concerns, die Verwaltung komplexer Datenfluss-Graphen und die Koordination asynchroner Operationen innerhalb einer vorhersagbaren, sequenziellen Event-Pipeline.
Collapses redundant data requests into single computation nodes to protect underlying sources and improve performance.
Wretch is a chainable HTTP client library and Fetch API wrapper designed to simplify network request configuration, response parsing, and error handling. It functions as a network request manager and middleware framework for managing the end-to-end lifecycle of HTTP calls. The project is distinguished by a fluent interface for request configuration and a modular extension system. It utilizes a programmable response resolver to automate parsing and a plugin-based architecture to add new capabilities to request and response objects. The library covers a broad range of operational capabilities,
Collapses multiple identical outgoing requests into a single execution to prevent redundant network calls.
orpc is a contract-first API development framework for TypeScript that starts with a shared contract definition and generates type-safe clients and servers from that single source of truth. It guarantees end-to-end type safety, meaning inputs, outputs, errors, and streaming data are all checked at compile time across the client–server boundary. What distinguishes orpc from typical RPC frameworks is its ability to export contracts as OpenAPI specifications, to optimize server-side rendering by calling API handlers directly inside the server process, and to support real‑time bidirectional commu
Prevents redundant client-side requests by grouping similar ones so only one is sent to the server.
One ist ein Full-Stack, plattformübergreifendes React-Framework und eine Multi-Target-Rendering-Engine, die darauf ausgelegt ist, Web- und native Anwendungen aus einer einzigen Codebasis zu erstellen. Es nutzt eine Vite-basierte Build-Pipeline zur Verwaltung gemeinsam genutzter Komponenten und bietet einen typsicheren Dateisystem-Router, der Navigationsdefinitionen automatisch generiert. Das Framework zeichnet sich durch die Unterstützung einer hybriden Rendering-Strategie aus, die es ermöglicht, einzelne Routen als statische Seiten, serverseitig gerenderte Seiten oder Single-Page-Anwendungen zu konfigurieren. Es verwendet eine plattformspezifische Logikauflösung und ein gemeinsames Vite-Plugin, um die Kompilierung für Web- und native Zielumgebungen zu koordinieren. Das System deckt ein breites Spektrum an Funktionen ab, darunter serverseitige Datenlader, API-Routenverwaltung und Middleware-basierte Request-Interception. Es bietet Tools für die Integration nativer Gerätefunktionen und Bridge-basierte Layouts für die mobile Navigation sowie Performance-Optimierungen wie parallele statische Generierung und Asset-Preloading. Anwendungen können auf Produktionsservern oder serverlosen Edge-Plattformen wie Vercel und Cloudflare bereitgestellt werden.
Implements request collapsing to share results of simultaneous calls to the same route and reduce server load.
Dieses Projekt ist ein chinesisches Lokalisierungs-Repository und technisches Übersetzungsprojekt, das darauf ausgelegt ist, prägnante Programmierprojekte und technische Dokumentationen für chinesischsprachige Personen zugänglich zu machen. Es bietet eine Sammlung übersetzter Ressourcen und kuratierter Mappings von Informatik-Terminologie, um eine konsistente Übersetzung technischer Konzepte zu gewährleisten. Das Projekt implementiert einen Software-Lokalisierungs-Workflow, der englischsprachige technische Leitfäden und Codebase-Dokumentationen in das Chinesische konvertiert. Dieser Prozess nutzt eine technische Glossar-Ressource und ein ressourcengesteuertes Lokalisierungsmodell, um die ursprüngliche Bedeutung und den technischen Kontext des Quellmaterials beizubehalten. Das Repository deckt die Übersetzung technischer Dokumentationen und die Verwaltung standardisierter Übersetzungs-Mappings ab, um eine konsistente technische Wortwahl über diverse Softwareprojekte hinweg sicherzustellen.
Filters out previously visited nodes during graph traversal to prevent redundant results.
Apache AGE is a graph database extension for PostgreSQL that adds openCypher graph query capabilities directly within the relational database environment. It functions as a loadable extension that translates Cypher graph traversal queries into SQL expressions, enabling users to run pattern matching and path analysis alongside standard SQL operations within a single database instance. The extension stores labeled, directed property graphs as isolated schemas with internal relational tables for vertices, edges, and labels, preventing cross-graph interference. It supports hybrid query execution
Supports removing duplicate rows from graph query results using the DISTINCT modifier.
PINRemoteImage ist ein Bild-Download- und Caching-Framework für iOS. Es fungiert als Remote-Image-Loader und Multi-Threaded-Manager, der darauf ausgelegt ist, Bilder von URLs abzurufen und in Benutzeroberflächen zu rendern. Das Projekt enthält spezialisierte Decoder zum Rendern von GIF- und animierten WebP-Formaten. Es verfügt zudem über einen progressiven JPG-Renderer, der Scans mit Weichzeichner-Effekten anzeigt, um die wahrgenommene Ladegeschwindigkeit bei langsamen Verbindungen zu verbessern. Die Bibliothek verwaltet gleichzeitige Download-Aufgaben mit Request-Deduplizierung und nutzt ein hybrides Speicher- und Disk-Caching-System. Sie handhabt die Skalierung hochauflösender Assets und führt die Bilddekodierung auf Hintergrund-Threads aus, um die Reaktionsfähigkeit der Benutzeroberfläche aufrechtzuerhalten.
Collapses multiple concurrent network requests for the same image resource to prevent redundant data transfers.