Awesome GitHub RepositoriesConcurrent Request Pooling

Guzzle is a PHP HTTP client used for sending synchronous and asynchronous requests to web services. It serves as a concurrent HTTP request manager, an HTTP stream handler, and a middleware-based HTTP pipeline. The project is a PSR-7 compliant client, utilizing standardized PHP interfaces for requests, responses, and streams. The library differentiates itself through a customizable functional handler stack that allows for the interception and modification of the request and response lifecycle. It features an adapter-based transport system that enables swapping between network implementations,

This project is a comprehensive educational guide and framework for building web scrapers using Python. It provides a course-based approach to data extraction, combining a Python crawler framework with tutorials on web reverse engineering and network traffic analysis. The project distinguishes itself by covering advanced extraction challenges, including the decryption of obfuscated JavaScript and the bypass of anti-scraping measures. It specifically addresses mobile application scraping through the simulation of user interactions and the interception of network traffic. The capability surfac

Puma is a concurrent HTTP server for Ruby applications that implements the Rack interface. It operates as a clustered web server, using a combination of worker processes and threads to handle multiple simultaneous web connections via TCP ports or UNIX domain sockets. The server features a master-worker process model that utilizes multiple CPU cores and employs copy-on-write preloading to reduce memory usage. It supports zero-downtime restarts through socket-handover capabilities, allowing application updates without dropping pending network requests. The project includes a token-authenticate

This project is a JavaScript-based HTTP download manager and accelerator designed to increase file transfer rates within a web browser. It functions as a multi-threaded download accelerator that retrieves remote resources by splitting files into smaller segments for simultaneous downloading. The tool utilizes HTTP range requests to fetch multiple file segments in parallel, maximizing available network bandwidth. It manages this process through a client-side blob buffer and memory management system that stores binary data before assembling the segments into the final file. The system covers c

Iron es un framework web de Rust utilizado para construir aplicaciones web concurrentes y APIs. Funciona como un servidor HTTP concurrente y proporciona un despachador de rutas sin estado para mapear rutas de URL entrantes y globs a funciones de controlador específicas. El proyecto se centra en un pipeline de solicitudes basado en middleware, que permite que el ciclo de solicitud-respuesta se extienda mediante plugins y modificadores. Utiliza un contenedor de estado seguro para hilos para almacenar memoria de aplicación compartida accesible a través de todos los controladores y middlewares concurrentes. El framework cubre amplias áreas de capacidad, incluyendo enrutamiento dinámico de API, alojamiento de archivos estáticos y gestión del estado de sesiones web. También incluye herramientas para registrar el tráfico HTTP y analizar cuerpos de solicitud y parámetros de URL.

Faraday is an HTTP client library for Ruby that sends requests and processes responses through a middleware pipeline with pluggable adapters. Its core identity is built around a middleware-pipeline architecture where HTTP requests and responses flow through a chain of components that can modify, log, or transform data before reaching the backend, combined with an adapter-based backend abstraction that delegates HTTP execution to interchangeable backends like Net::HTTP or Typhoeus. The library distinguishes itself through a parallel-execution engine that dispatches multiple HTTP requests concu

Trafilatura is a Python library and command-line tool for extracting clean, structured text and metadata from web pages. It downloads HTML content, identifies the main body of text, and strips away navigation, ads, and other boilerplate, returning the core article content along with fields like title, author, date, and URL. The tool can also extract user comments and test whether a page contains extractable text, making it a general-purpose web text extraction library. What distinguishes Trafilatura from simpler extractors is its configurable extraction pipeline, which offers high-speed, high

Hakrawler is a command-line web spider tool designed for security reconnaissance, built to crawl target websites and extract hyperlinks along with JavaScript file references. As a focused reconnaissance utility, it collects every discoverable URL and script source from a given domain, mapping the attack surface for penetration testing and vulnerability assessment. The tool differentiates itself through its concurrent architecture: a fixed-size goroutine pool fetches pages in parallel, while CSS selectors parse HTML to extract anchor and script references. A depth-aware recursion limiter preve

resumable.js es una biblioteca de JavaScript para gestionar la carga de archivos grandes utilizando la API de archivos de HTML5. Funciona como un transmisor de datos fragmentados y gestor de cargas reanudables, dividiendo los archivos en segmentos más pequeños para asegurar una entrega confiable al servidor remoto. La biblioteca se distingue por su capacidad para recuperar el progreso de la carga tras interrupciones de red o reinicios del navegador. Esto se logra mediante la transferencia de datos reanudable y la verificación de fragmentos en el servidor, que comprueba los segmentos existentes para evitar la transmisión redundante de datos. El sistema gestiona cargas de archivos concurrentes y colas de solicitudes para optimizar el ancho de banda. Incluye capacidades para el control del estado de carga —permitiendo a los usuarios pausar, reanudar o cancelar transferencias— y seguimiento del progreso basado en eventos para archivos individuales y lotes. Además, proporciona primitivas para vincular la captura de archivos a elementos HTML mediante botones de exploración e interfaces de arrastrar y soltar.

Grequests es un batcher HTTP asíncrono y una librería cliente basada en Gevent utilizada para ejecutar grandes conjuntos de peticiones de red simultáneamente. Funciona como un wrapper de peticiones concurrentes para la librería Requests, permitiendo operaciones no bloqueantes para reducir el tiempo total de espera de las respuestas del servidor. El proyecto proporciona un modelo de ejecución de pool de tareas para manejar operaciones de red por lotes, como web scraping de alto rendimiento y sondeo de APIs. Puede transmitir respuestas a medida que llegan mediante un generador, permitiendo el procesamiento inmediato de datos sin esperar a que todo el lote se complete. La librería cubre áreas de capacidad amplias, incluyendo la gestión de peticiones HTTP concurrentes y el manejo de excepciones de petición durante la ejecución por lotes.

Waybackurls is a command-line OSINT tool that retrieves every known URL for a given domain from the Wayback Machine archive. It functions as a domain reconnaissance utility, discovering forgotten API endpoints, legacy pages, and hidden files by querying the public web archive API. The tool processes domains independently and statelessly, reading domain names from standard input and streaming discovered URLs line-by-line to standard output. This design enables seamless integration into Unix command pipelines, allowing users to chain waybackurls with other tools for filtering, sorting, and furt

Typhoeus is a Ruby wrapper for libcurl that functions as a session-based HTTP client. It provides an interface for making both synchronous and asynchronous network requests. The project acts as a parallel request manager, using a managed queue to execute multiple network requests concurrently. It further distinguishes itself as a mocking tool for stubbing requests with predefined responses and as a caching layer that stores responses to avoid redundant network calls. The library covers a broad range of capabilities including session cookie management, response body streaming for large files,

ShuiZe_0x727 es un framework de recolección de inteligencia de código abierto y herramienta de gestión de superficie de ataque. Funciona como un motor de descubrimiento de activos y agregador de ciberinteligencia diseñado para identificar activos expuestos a internet, mapear infraestructura de red y visualizar la exposición total de la red. El proyecto integra escaneo de vulnerabilidades y detección de fugas de datos sensibles para identificar debilidades de seguridad y puntos de acceso no autorizados. Emplea una combinación de consultas a APIs de espacio de red, análisis de logs de certificados y escaneo de repositorios públicos para extraer credenciales filtradas, claves de API y rutas administrativas internas. El framework proporciona capacidades para la recolección automatizada de información y la investigación de ciberinteligencia, utilizando un motor de escaneo basado en plugins para detectar vulnerabilidades en servicios web y puertos abiertos. Los datos de activos recopilados y los hallazgos de seguridad se exportan a hojas de cálculo formateadas para su análisis y auditoría offline.

Ali is an HTTP load testing tool and traffic generator used to measure system performance and stability by sending high volumes of requests to a target URL. It functions as a performance metrics exporter and real-time visualizer, simulating client behavior through configurable request bodies, custom headers, and HTTP/2 support. The project provides real-time performance monitoring via interactive charts that plot latency and request percentiles as data is collected. It identifies system bottlenecks through a combination of live performance plotting and the exportation of raw data points and a

JobSpy is a job board scraper and listing aggregator designed to extract employment opportunities from multiple websites and compile them into a unified dataset. It functions as a job search automation tool that programmatically collects vacancies based on keywords, locations, and specific filters. The project serves as a web scraping framework that utilizes proxy routing and user-agent rotation to bypass rate limits and avoid server-side blocking during data extraction. It includes infrastructure for concurrent request aggregation and schema-based data normalization to ensure consistent form