awesome-repositories.com
© 2026 Bringes Technology SRL·VAT RO45896025·hello@bringes.io
MCPSitemapPrivacyTerms
Web Crawling Infrastructure · Awesome GitHub Repositories

1 repo

Awesome GitHub RepositoriesWeb Crawling Infrastructure

Foundational software and environment configurations required to support and maintain web data collection operations.

Explore 1 awesome GitHub repository matching web development · Web Crawling Infrastructure. Refine with filters or upvote what's useful.

  1. Home
  2. Web Development
  3. Web Automation and Scraping
  4. Web Scraping and Automation
  5. Web Crawling Infrastructure

Awesome Web Crawling Infrastructure GitHub Repositories

Describe the repository you're looking for…
We'll search the best matching repositories with AI.
  • unclecode/crawl4ai

    unclecode/crawl4ai

    60,452GitHubView on GitHub↗

    Crawl4AI is an AI-powered web crawling and data extraction engine designed to transform complex web content into structured formats. It functions as a headless browser orchestrator, enabling the navigation of dynamic websites, the execution of custom scripts, and the capture of visual assets like screenshots and PDFs.

    Python

Explore sub-tags

  • Crawling Environment ConfigurationsAutomated setup and dependency management for browser automation and data collection environments.