awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
brendonboshell avatar

brendonboshell/supercrawler

0
View on GitHub↗

Supercrawler

Supercrawler is a Node.js web crawler. It is designed to be highly configurable and easy to use.

Features

  • JavaScript Crawling Frameworks - Crawler with custom handlers and rate limiting.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI
  • Web Scraping - Crawler with support for robots.txt and rate limiting.
  • 381 stars·63 forks·JavaScript·Apache-2.0·12 views

    Star history

    Star history chart for brendonboshell/supercrawlerStar history chart for brendonboshell/supercrawler

    How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

    Frequently asked questions

    What does brendonboshell/supercrawler do?

    Supercrawler is a Node.js web crawler. It is designed to be highly configurable and easy to use.

    What are the main features of brendonboshell/supercrawler?

    The main features of brendonboshell/supercrawler are: JavaScript Crawling Frameworks, Web Scraping.

    Which projects share features with brendonboshell/supercrawler?

    Projects with overlapping indexed features include: ionicabizau/scrape-it — scrape-it is a Node.js web scraper and HTML parser designed to extract structured data from websites and HTML files.… lapwinglabs/x-ray — X-Ray is a web scraping framework and asynchronous web crawler designed to extract structured data from websites. It… antivanov/js-crawler — js-crawler. cgiffard/node-simplecrawler — simplecrawler is designed to provide a basic, flexible and robust API for crawling websites. It was written to… bda-research/node-crawler — node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It… martinsbalodis/web-scraper-chrome-extension — Web Scraper is a chrome browser extension built for data extraction from web pages. Using this extension you can…

    Projects sharing features with Supercrawler

    These projects share indexed features with Supercrawler. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
    • bda-research/node-crawlerbda-research avatar

      bda-research/node-crawler

      6,785View on GitHub↗

      node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It functions as a rate-limited HTTP client and a headless HTML parser, providing the infrastructure to visit large sets of URLs asynchronously while preventing duplicate processing through task deduplication. The project distinguishes itself through a proxy rotation manager that cycles user agents and proxy servers to bypass access restrictions. It utilizes the HTTP/2 protocol to improve request performance and server compatibility during large-scale scraping operations. The syst

      TypeScriptcheeriocrawlerextract-data
      View on GitHub↗6,785
    • cgiffard/node-simplecrawlercgiffard avatar

      cgiffard/node-simplecrawler

      2,133View on GitHub↗

      simplecrawler is designed to provide a basic, flexible and robust API for crawling websites. It was written to archive, analyse, and search some very large websites and has happily chewed through hundreds of thousands of pages and written tens of gigabytes to disk without issue.

      JavaScript
      View on GitHub↗2,133
    • antivanov/js-crawlerantivanov avatar

      antivanov/js-crawler

      257View on GitHub↗

      js-crawler

      TypeScript
      View on GitHub↗257
    ionicabizau/scrape-itIonicaBizau avatar

    IonicaBizau/scrape-it

    4,069View on GitHub↗

    scrape-it is a Node.js web scraper and HTML parser designed to extract structured data from websites and HTML files. It functions as a web data extraction tool that retrieves specific information from DOM elements and converts web content into usable data fields. The tool uses CSS selectors to target specific data points and employs schema-driven data mapping to organize unstructured web text into a consistent format. It supports custom value transformation to convert raw extracted strings into specific data formats. The system provides capabilities for web data extraction and automated cont

    JavaScripthacktoberfestnode-scraperscraper
    View on GitHub↗4,069
    Compare all 30 related projects→