awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
n0tan3rd avatar

n0tan3rd/squidwarc

0
View on GitHub↗
176 stars·25 forks·JavaScript·Apache-2.0·12 viewsn0tan3rd.github.io/Squidwarc↗

Squidwarc

Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head.

Features

  • JavaScript Crawling Frameworks - High-fidelity archival crawler using Chrome.
  • Web Scraping - High-fidelity archival crawler using headless Chrome.

Star history

Star history chart for n0tan3rd/squidwarcStar history chart for n0tan3rd/squidwarc

How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Projects sharing features with Squidwarc

These projects share indexed features with Squidwarc. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
  • bda-research/node-crawlerbda-research avatar

    bda-research/node-crawler

    6,785View on GitHub↗

    node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It functions as a rate-limited HTTP client and a headless HTML parser, providing the infrastructure to visit large sets of URLs asynchronously while preventing duplicate processing through task deduplication. The project distinguishes itself through a proxy rotation manager that cycles user agents and proxy servers to bypass access restrictions. It utilizes the HTTP/2 protocol to improve request performance and server compatibility during large-scale scraping operations. The syst

    TypeScriptcheeriocrawlerextract-data
    View on GitHub↗6,785
  • brendonboshell/supercrawlerbrendonboshell avatar

    brendonboshell/supercrawler

    381View on GitHub↗

    Supercrawler is a Node.js web crawler. It is designed to be highly configurable and easy to use.

    JavaScript
    View on GitHub↗381
  • antivanov/js-crawlerantivanov avatar

    antivanov/js-crawler

    257View on GitHub↗

    js-crawler

    TypeScript
    View on GitHub↗257
  • cgiffard/node-simplecrawlercgiffard avatar

    cgiffard/node-simplecrawler

    2,133View on GitHub↗

    simplecrawler is designed to provide a basic, flexible and robust API for crawling websites. It was written to archive, analyse, and search some very large websites and has happily chewed through hundreds of thousands of pages and written tens of gigabytes to disk without issue.

    JavaScript
    View on GitHub↗2,133
Compare all 30 related projects→

Frequently asked questions

What does n0tan3rd/squidwarc do?

Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head.

What are the main features of n0tan3rd/squidwarc?

The main features of n0tan3rd/squidwarc are: JavaScript Crawling Frameworks, Web Scraping.

Which projects share features with n0tan3rd/squidwarc?

Projects with overlapping indexed features include: cgiffard/node-simplecrawler — simplecrawler is designed to provide a basic, flexible and robust API for crawling websites. It was written to… ionicabizau/scrape-it — scrape-it is a Node.js web scraper and HTML parser designed to extract structured data from websites and HTML files.… antivanov/js-crawler — js-crawler. brendonboshell/supercrawler — Supercrawler is a Node.js web crawler. It is designed to be highly configurable and easy to use. bda-research/node-crawler — node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It… lapwinglabs/x-ray — X-Ray is a web scraping framework and asynchronous web crawler designed to extract structured data from websites. It…