Awesome GitHub RepositoriesConcurrent Request Pooling

Guzzle is a PHP HTTP client used for sending synchronous and asynchronous requests to web services. It serves as a concurrent HTTP request manager, an HTTP stream handler, and a middleware-based HTTP pipeline. The project is a PSR-7 compliant client, utilizing standardized PHP interfaces for requests, responses, and streams. The library differentiates itself through a customizable functional handler stack that allows for the interception and modification of the request and response lifecycle. It features an adapter-based transport system that enables swapping between network implementations,

This project is a comprehensive educational guide and framework for building web scrapers using Python. It provides a course-based approach to data extraction, combining a Python crawler framework with tutorials on web reverse engineering and network traffic analysis. The project distinguishes itself by covering advanced extraction challenges, including the decryption of obfuscated JavaScript and the bypass of anti-scraping measures. It specifically addresses mobile application scraping through the simulation of user interactions and the interception of network traffic. The capability surfac

Puma is a concurrent HTTP server for Ruby applications that implements the Rack interface. It operates as a clustered web server, using a combination of worker processes and threads to handle multiple simultaneous web connections via TCP ports or UNIX domain sockets. The server features a master-worker process model that utilizes multiple CPU cores and employs copy-on-write preloading to reduce memory usage. It supports zero-downtime restarts through socket-handover capabilities, allowing application updates without dropping pending network requests. The project includes a token-authenticate

This project is a JavaScript-based HTTP download manager and accelerator designed to increase file transfer rates within a web browser. It functions as a multi-threaded download accelerator that retrieves remote resources by splitting files into smaller segments for simultaneous downloading. The tool utilizes HTTP range requests to fetch multiple file segments in parallel, maximizing available network bandwidth. It manages this process through a client-side blob buffer and memory management system that stores binary data before assembling the segments into the final file. The system covers c

Iron is a Rust web framework used for building concurrent web applications and APIs. It functions as a concurrent HTTP server and provides a stateless route dispatcher to map incoming URL paths and globs to specific handler functions. The project is centered around a middleware-based request pipeline, which allows the request-response cycle to be extended via plugins and modifiers. It utilizes a thread-safe state container to store shared application memory accessible across all concurrent handlers and middleware. The framework covers broad capability areas including dynamic API routing, sta

Faraday is an HTTP client library for Ruby that sends requests and processes responses through a middleware pipeline with pluggable adapters. Its core identity is built around a middleware-pipeline architecture where HTTP requests and responses flow through a chain of components that can modify, log, or transform data before reaching the backend, combined with an adapter-based backend abstraction that delegates HTTP execution to interchangeable backends like Net::HTTP or Typhoeus. The library distinguishes itself through a parallel-execution engine that dispatches multiple HTTP requests concu

Trafilatura is a Python library and command-line tool for extracting clean, structured text and metadata from web pages. It downloads HTML content, identifies the main body of text, and strips away navigation, ads, and other boilerplate, returning the core article content along with fields like title, author, date, and URL. The tool can also extract user comments and test whether a page contains extractable text, making it a general-purpose web text extraction library. What distinguishes Trafilatura from simpler extractors is its configurable extraction pipeline, which offers high-speed, high

Hakrawler is a command-line web spider tool designed for security reconnaissance, built to crawl target websites and extract hyperlinks along with JavaScript file references. As a focused reconnaissance utility, it collects every discoverable URL and script source from a given domain, mapping the attack surface for penetration testing and vulnerability assessment. The tool differentiates itself through its concurrent architecture: a fixed-size goroutine pool fetches pages in parallel, while CSS selectors parse HTML to extract anchor and script references. A depth-aware recursion limiter preve

resumable.js 是一个使用 HTML5 File API 管理大文件上传的 JavaScript 库。它充当分块数据传输器和断点续传管理器，将文件分割成较小的片段，以确保可靠地传输到远程服务器。 该库的独特之处在于能够在网络中断或浏览器重启后恢复上传进度。它通过断点续传和服务器端分块验证来实现这一点，通过检查服务器上已存在的片段来避免冗余数据传输。 系统管理并发文件上传和请求队列以优化带宽。它包括上传状态控制功能（允许用户暂停、恢复或取消传输），以及针对单个文件和批次的事件驱动进度追踪。此外，它还提供了通过浏览按钮和拖拽界面将文件捕获绑定到 HTML 元素的基元。

Grequests 是一个异步 HTTP 批处理程序和基于 Gevent 的客户端库，用于同时执行大量网络请求。它作为 Requests 库的并发请求包装器，支持非阻塞操作，以减少等待服务器响应的总时间。 该项目提供任务池执行模型来处理批处理网络操作，例如高吞吐量网页抓取和 API 轮询。它可以通过生成器在响应到达时进行流式传输，从而无需等待整个批次完成即可进行即时数据处理。 该库涵盖广泛的功能领域，包括并发 HTTP 请求管理以及批处理执行期间的请求异常处理。

Waybackurls is a command-line OSINT tool that retrieves every known URL for a given domain from the Wayback Machine archive. It functions as a domain reconnaissance utility, discovering forgotten API endpoints, legacy pages, and hidden files by querying the public web archive API. The tool processes domains independently and statelessly, reading domain names from standard input and streaming discovered URLs line-by-line to standard output. This design enables seamless integration into Unix command pipelines, allowing users to chain waybackurls with other tools for filtering, sorting, and furt

Typhoeus is a Ruby wrapper for libcurl that functions as a session-based HTTP client. It provides an interface for making both synchronous and asynchronous network requests. The project acts as a parallel request manager, using a managed queue to execute multiple network requests concurrently. It further distinguishes itself as a mocking tool for stubbing requests with predefined responses and as a caching layer that stores responses to avoid redundant network calls. The library covers a broad range of capabilities including session cookie management, response body streaming for large files,

ShuiZe_0x727 是一个开源情报收集框架和攻击面管理工具。它作为一个资产发现引擎和网络情报聚合器，旨在识别面向互联网的资产、映射网络基础设施并可视化总网络暴露情况。 该项目集成了漏洞扫描和敏感数据泄露检测，以识别安全弱点和未经授权的访问点。它结合了网络空间 API 查询、证书日志分析和公共存储库扫描，以提取泄露的凭据、API 密钥和内部管理路径。 该框架提供了自动化信息收集和网络情报研究的功能，利用基于插件的扫描引擎来检测跨 Web 服务和开放端口的漏洞。收集到的资产数据和安全发现被导出为格式化的电子表格，以便进行离线分析和审计。

Ali 是一个 HTTP 负载测试工具和流量生成器，用于通过向目标 URL 发送大量请求来衡量系统性能和稳定性。它作为一个性能指标导出器和实时可视化工具，通过可配置的请求体、自定义标头和 HTTP/2 支持来模拟客户端行为。 该项目通过交互式图表提供实时性能监控，在收集数据时绘制延迟和请求百分位数。它通过实时性能绘图与将原始数据点和聚合摘要导出到文件以进行事后分析相结合，识别系统瓶颈。 该工具涵盖了流量管理的广泛能力，包括控制网络行为（如保持连接和请求超时设置）。它进一步支持通过 HTTP 方法和有效负载的参数驱动配置进行 API 压力测试。

JobSpy is a job board scraper and listing aggregator designed to extract employment opportunities from multiple websites and compile them into a unified dataset. It functions as a job search automation tool that programmatically collects vacancies based on keywords, locations, and specific filters. The project serves as a web scraping framework that utilizes proxy routing and user-agent rotation to bypass rate limits and avoid server-side blocking during data extraction. It includes infrastructure for concurrent request aggregation and schema-based data normalization to ensure consistent form