awesome-repositories.com
© 2026 Bringes Technology SRL·VAT RO45896025·hello@bringes.io
MCPSitemapPrivacyTerms
Distributed Crawling Systems · Awesome GitHub Repositories

1 repo

Awesome GitHub RepositoriesDistributed Crawling Systems

Self-hosted environments that manage asynchronous job queues and browser resources for large-scale data collection.

Explore 1 awesome GitHub repository matching web development · Distributed Crawling Systems. Refine with filters or upvote what's useful.

  1. Home
  2. Web Development
  3. Web Automation and Scraping
  4. Web Scraping and Automation
  5. Web Scraping Infrastructure
  6. Distributed Crawling Systems

Awesome Distributed Crawling Systems GitHub Repositories

Describe the repository you're looking for…
We'll search the best matching repositories with AI.
  • unclecode/crawl4ai

    unclecode/crawl4ai

    60,452GitHubView on GitHub↗

    Crawl4AI is an AI-powered web crawling and data extraction engine designed to transform complex web content into structured formats. It functions as a headless browser orchestrator, enabling the navigation of dynamic websites, the execution of custom scripts, and the capture of visual assets like screenshots and PDFs.

    Python