kennethreitz/grequests
Grequests
Grequests ist ein asynchroner HTTP-Batcher und eine auf Gevent basierende Client-Bibliothek, die verwendet wird, um große Mengen von Netzwerkanfragen gleichzeitig auszuführen. Er fungiert als Wrapper für gleichzeitige Anfragen für die Requests-Bibliothek und ermöglicht nicht-blockierende Operationen, um die Wartezeit auf Serverantworten zu reduzieren.
Das Projekt bietet ein Task-Pool-Ausführungsmodell für Batch-Netzwerkoperationen, wie z. B. hochperformantes Web Scraping und API-Polling. Es kann Antworten über einen Generator streamen, sobald sie eintreffen, was eine sofortige Datenverarbeitung ermöglicht, ohne auf den Abschluss des gesamten Batches warten zu müssen.
Die Bibliothek deckt breite Funktionsbereiche ab, einschließlich der Verwaltung gleichzeitiger HTTP-Anfragen und der Behandlung von Request-Exceptions während der Batch-Ausführung.
Features
- Parallel Request Executions - Sends multiple HTTP requests concurrently using a task pool to reduce total wait time compared to sequential processing.
- Greenlet-Based Concurrency - Uses lightweight greenlets to allow multiple network operations to run concurrently within a single operating system thread.
- Asynchronous HTTP Clients - Functions as a library for sending asynchronous HTTP requests and receiving data via non-blocking APIs.
- Batch Network Operations - Executes large sets of HTTP requests and processes the responses as a stream or collective group.
- HTTP Client Libraries - Implements a client library for making concurrent and asynchronous HTTP requests using the Gevent framework.
- Coroutine-Based Asynchronous I/O - Implements asynchronous I/O using Gevent coroutines to manage high concurrency without blocking threads.
- Concurrent Request Pools - Implements capped resource pools to throttle and manage a large volume of simultaneous asynchronous network requests.
- Concurrent Request Pooling - Sends multiple network requests at once using Gevent and capped pools to complete large workloads faster.
- Web Scraping - Gathers data from many web pages simultaneously by executing non-blocking requests in a task pool.
- Response Streaming - Provides the ability to process HTTP response bodies as continuous data streams via a generator to optimize memory and latency.
- Socket Monkeypatching - Replaces standard library blocking calls with non-blocking versions to prevent the event loop from freezing.
- API Polling - Enables simultaneous fetching of data from multiple API endpoints to reduce total server wait time.
- Synchronous-Style Asynchronous APIs - Wraps non-blocking network calls in a familiar sequential syntax while executing them asynchronously.
- High-Volume Web Scrapers - Provides a high-throughput networking tool for large-scale data collection from multiple web pages without sequential waiting.
- HTTP Clients - Asynchronous HTTP requests using gevent.
