awesome-repositories.com
© 2026 Bringes Technology SRL·VAT RO45896025·hello@bringes.io
MCPSitemapPrivacyTerms
Crawling Optimization · Awesome GitHub Repositories

1 repo

Awesome GitHub RepositoriesCrawling Optimization

Techniques and configurations for managing memory, request rates, and concurrency in large-scale data collection tasks.

Explore 1 awesome GitHub repository matching web development · Crawling Optimization. Refine with filters or upvote what's useful.

  1. Home
  2. Web Development
  3. Web Automation and Scraping
  4. Web Scraping and Automation
  5. Web Scraping
  6. Crawling Optimization

Awesome Crawling Optimization GitHub Repositories

Describe the repository you're looking for…
We'll search the best matching repositories with AI.
  • scrapy/scrapy

    scrapy/scrapy

    59,824GitHubView on GitHub↗

    Scrapy is a comprehensive framework designed for automated web data extraction and large-scale crawling. It operates on an asynchronous, event-driven engine that manages non-blocking network requests and data processing tasks, allowing for the efficient retrieval of structured information from web documents using path-

    Pythoncrawlercrawlingframework