Awesome GitHub RepositoriesAPI Request Deduplication

Parse Server is a backend-as-a-service solution and Node.js framework that provides a ready-to-use REST and GraphQL API for mobile and web applications. It functions as a core backend infrastructure for managing database schemas, user authentication, and API routing. The system distinguishes itself with a real-time data engine that pushes database updates to clients via WebSockets and a GraphQL server that automatically generates schemas based on application data models. It also features an adapter-based storage layer that abstracts interactions with various cloud and local backends. The pla

Cortex is an open-source, horizontally scalable metrics platform that ingests, stores, and queries Prometheus-compatible time-series data with multi-tenant isolation. It accepts metrics via Prometheus remote write and OpenTelemetry, executes PromQL queries against both recent and historical data, and provides a Prometheus-compatible alerting and recording rule engine with an integrated Alertmanager. The system is built as a set of independently scalable microservices that use hash-ring-based sharding, gossip-based cluster membership, and tenant-aware object storage to distribute workloads acro

Scrapy-Redis is a library that transforms Scrapy into a distributed web crawling framework by replacing its in-memory scheduler with a Redis-backed component. This allows multiple Scrapy spider workers to coordinate through a shared request queue, enabling them to consume URLs concurrently while a Redis set tracks seen URLs across all workers to prevent duplicate crawls. The system persists crawl state—including pending requests and already-crawled URLs—in Redis, so a paused or crashed spider can resume from where it left off without losing progress. The library provides a Redis-based duplica

这是一个分布式网络爬取框架，支持抓取任务的水平扩展。它使用 Redis 作为集中式请求队列管理器和状态存储，以协调跨多个服务器实例的爬取进度和请求元数据。 该系统通过共享单个请求队列来分发爬取工作负载，并利用分布式重复过滤器来防止多个工作节点访问同一页面。它将复杂的请求状态和元数据作为 JSON 字符串持久化在共享远程存储中。 该框架还通过将抓取到的项目推送到共享队列以供单独的处理工作节点并行消费，从而提供分布式数据处理功能。

Blackbird is an open-source OSINT investigation tool that searches across hundreds of online platforms to discover accounts linked to a given username or email address. It functions as a username and email search engine, consolidating discovered profiles into a single list with low false positives for investigative analysis. The tool incorporates an AI-enhanced profile analyzer that uses a built-in AI API to generate behavioral and technical summaries of discovered online profiles. It also provides a documentation query interface that accepts natural-language questions via HTTP GET requests t

Libpostal 是一个专为国际地址解析和标准化设计的 C 库。它利用统计自然语言处理（NLP）和语言分类器，将非结构化的全球地址字符串分解为结构化组件，并通过扩展缩写和解决跨语言的区域命名差异来标准化街道地址。 该项目提供文本转写工具，将各种脚本转换为标准化的拉丁-ASCII 或 NFD 形式。它还包括地址去重功能，使用对称模糊匹配来识别不同的地址记录是否指向同一个物理位置。 该库涵盖了更广泛的文本处理需求，例如 UTF-8 标准化，以及将书写数字和罗马数字转换为标准数字表示。它允许通过外部配置文件扩展地址识别功能，以添加新的语言和同义词。

Dedupe is a machine learning tool for entity resolution that identifies and merges duplicate records in structured datasets. It uses active learning to train a matching model from human-labeled examples, learning which field-level similarities are most important for detecting duplicates without requiring manual rule writing. The system combines fingerprint-based blocking to reduce pairwise comparisons, enabling efficient matching on large datasets, and groups scored record pairs into clusters using a configurable similarity threshold. The tool provides multiple interfaces for different workfl

CrawlerTutorial 是一个全面的 Python 网络爬虫教程和框架，旨在从静态和动态网站中提取数据。它作为一个网络数据提取管道和 HTTP 请求编排器，涵盖了从初始获取到最终数据存储的爬虫应用程序全生命周期。 该项目提供了关于反机器人绕过技术和 Web API 逆向工程的专业指导。它包括通过身份掩码和代理轮换规避浏览器检测的方法，以及通过分析网络流量和请求签名识别隐藏 API 端点的技术。 该框架包含广泛的功能，包括针对 JavaScript 重度页面的浏览器自动化、通过 QR 码或短信的自动用户身份验证以及会话持久性管理。它还具有用于清理原始文本、删除重复记录并将收集到的信息持久化到平面文件或关系数据库中的数据预处理工具。

ClawRouter is an AI model router and API gateway designed to classify query complexity and assign prompts to the most efficient model tier. It operates as a multi-model AI proxy that orchestrates traffic between various large language models and AI media generators through a unified interface. The project distinguishes itself by integrating a non-custodial micropayment processor using the x402 protocol. This allows for per-request API access and USDC settlement on Base and Solana chains, replacing static API keys with wallet-based authentication and real-time budget enforcement. The system c