2 Repos
Awesome GitHub RepositoriesRetry Suppression Policies
Mechanisms for signaling the crawler to immediately halt retries for specific non-recoverable errors.
Distinct from Request Retries: Distinct from general request retries: focuses on the logic to prevent retries, not the retry mechanism itself.
Explore 2 awesome GitHub repositories matching devops & infrastructure · Retry Suppression Policies. Refine with filters or upvote what's useful.
Awesome Retry Suppression Policies GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- apify/crawlee
apify/crawlee
24,002Auf GitHub ansehenCrawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob
Signals the crawler to stop attempting specific requests immediately when non-recoverable errors occur.
TypeScriptapifyautomationcrawler - apify/crawlee-python
apify/crawlee-python
8,097Auf GitHub ansehenCrawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra
Signals the crawler to halt retries for non-recoverable fatal errors to optimize processing.
Pythonapifyautomationbeautifulsoup
