kennethreitz/grequests
Grequests
Grequests is an asynchronous HTTP batcher and Gevent-based client library used to execute large sets of network requests simultaneously. It functions as a concurrent request wrapper for the Requests library, enabling non-blocking operations to reduce the total time spent waiting for server responses.
The project provides a task-pool execution model to handle batch network operations, such as high-throughput web scraping and API polling. It can stream responses as they arrive via a generator, allowing for immediate data processing without waiting for the entire batch to complete.
The library covers broad capability areas including concurrent HTTP request management and the handling of request exceptions during batch execution.
Features
- Parallel Request Executions - Sends multiple HTTP requests concurrently using a task pool to reduce total wait time compared to sequential processing.
- Greenlet-Based Concurrency - Uses lightweight greenlets to allow multiple network operations to run concurrently within a single operating system thread.
- Asynchronous HTTP Clients - Functions as a library for sending asynchronous HTTP requests and receiving data via non-blocking APIs.
- Batch Network Operations - Executes large sets of HTTP requests and processes the responses as a stream or collective group.
- HTTP Client Libraries - Implements a client library for making concurrent and asynchronous HTTP requests using the Gevent framework.
- Coroutine-Based Asynchronous I/O - Implements asynchronous I/O using Gevent coroutines to manage high concurrency without blocking threads.
- Concurrent Request Pools - Implements capped resource pools to throttle and manage a large volume of simultaneous asynchronous network requests.
- Concurrent Request Pooling - Sends multiple network requests at once using Gevent and capped pools to complete large workloads faster.
- Web Scraping - Gathers data from many web pages simultaneously by executing non-blocking requests in a task pool.
- Response Streaming - Provides the ability to process HTTP response bodies as continuous data streams via a generator to optimize memory and latency.
- Socket Monkeypatching - Replaces standard library blocking calls with non-blocking versions to prevent the event loop from freezing.
- API Polling - Enables simultaneous fetching of data from multiple API endpoints to reduce total server wait time.
- Synchronous-Style Asynchronous APIs - Wraps non-blocking network calls in a familiar sequential syntax while executing them asynchronously.
- High-Volume Web Scrapers - Provides a high-throughput networking tool for large-scale data collection from multiple web pages without sequential waiting.
- HTTP Clients - Asynchronous HTTP requests using gevent.
