spyoungtech/grequests
Grequests
Grequests is an asynchronous HTTP client and wrapper for the Requests library that uses Gevent coroutines to execute multiple network requests concurrently. It utilizes a non-blocking connection pool to manage simultaneous outgoing requests and improve throughput compared to sequential execution.
The library features an asynchronous response generator that yields HTTP responses as they complete, rather than waiting for an entire batch to finish. It provides a mechanism to limit the number of active connections to manage resource consumption and prevent destination servers from being overloaded.
The project covers capabilities for batch request management, high-volume data scraping, and concurrent API polling. It includes support for streaming large response bodies to maintain low memory usage and manages network timeouts and connection errors by injecting error states back into the response list.
Features
- Parallel Request Executions - Sends multiple HTTP requests concurrently using Gevent to reduce total wait time.
- Asynchronous HTTP Clients - Uses Gevent coroutines as an asynchronous HTTP client to execute concurrent network requests.
- Asynchronous Request Execution - Executes multiple network requests concurrently using non-blocking patterns to improve throughput.
- Concurrent Request Pools - Wraps the Requests library to manage pools of asynchronous connections for batch network operations.
- Request Batching - Manages groups of network requests and processes results using a generator as tasks complete.
- Concurrent Request Limits - Limits the number of simultaneous outgoing requests to manage resource consumption and prevent server overloading.
- Completion-Based Batch Generators - Provides a streaming interface that yields HTTP responses as they complete instead of waiting for the entire batch.
- Response Body Streaming - Handles massive HTTP response bodies through streaming to keep memory usage low.
- Response Streaming - Processes large HTTP response bodies as continuous data streams to minimize memory usage.
- Completion-Based Batch Generators - Yields responses as a generator as they complete, allowing processing without waiting for the entire batch.
- High Performance Scraping - Gathers information from many web pages quickly using a managed pool of non-blocking requests.
- API Polling - Fetches data from multiple API endpoints simultaneously with connection limits to avoid rate limits.
