1 dépôt
Awesome GitHub RepositoriesQueue State Configurations
Settings for managing the initialization and history of request queues.
Distinct from Task Queue Management: Distinct from general task queue management: focuses on crawl-specific queue history and retry state.
Explore 1 awesome GitHub repository matching devops & infrastructure · Queue State Configurations. Refine with filters or upvote what's useful.
Awesome Queue State Configurations GitHub Repositories
Trouvez les meilleurs dépôts grâce à l'IA.Nous recherchons les dépôts les plus pertinents grâce à l'IA.
- apify/crawlee
apify/crawlee
24,002Voir sur GitHubCrawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob
Allows configuring whether to clear request history or resume from previous states.
TypeScriptapifyautomationcrawler
