awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
CrawlScript avatar

CrawlScript/WebCollector

0
View on GitHub↗
3,091 stars·1,435 forks·Java·GPL-3.0·9 viewsgithub.com/CrawlScript/WebCollector↗

WebCollector

WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.

Features

  • Java Crawling Frameworks - Multi-threaded crawler with simple interfaces.

Star history

Star history chart for crawlscript/webcollectorStar history chart for crawlscript/webcollector

How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Frequently asked questions

What does crawlscript/webcollector do?

WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.

What are the main features of crawlscript/webcollector?

The main features of crawlscript/webcollector are: Java Crawling Frameworks.

Which projects share features with crawlscript/webcollector?

Projects with overlapping indexed features include: code4craft/webmagic — Webmagic is a Java web crawling framework designed for building scalable automated crawlers to download and process… digitalpebble/storm-crawler — A scalable, mature and versatile web crawler based on Apache Storm. internetarchive/heritrix3 — Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix… norconex/collector-http — Norconex HTTP Collector. pkwenda/webbee — 🐝 Web vertical crawler framework for fun. ssssssss-team/spider-flow — Spider-flow is a Java-based web crawling and data extraction platform that provides a centralized environment for…

Projects sharing features with WebCollector

These projects share indexed features with WebCollector. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
  • digitalpebble/storm-crawlerDigitalPebble avatar

    DigitalPebble/storm-crawler

    980View on GitHub↗

    A scalable, mature and versatile web crawler based on Apache Storm

    Java
    View on GitHub↗980
  • internetarchive/heritrix3internetarchive avatar

    internetarchive/heritrix3

    3,246View on GitHub↗

    Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect…

    Java
    View on GitHub↗3,246
  • norconex/collector-httpNorconex avatar

    Norconex/collector-http

    202View on GitHub↗

    Norconex HTTP Collector

    Java
    View on GitHub↗202
  • code4craft/webmagiccode4craft avatar

    code4craft/webmagic

    11,680View on GitHub↗

    Webmagic is a Java web crawling framework designed for building scalable automated crawlers to download and process large volumes of web pages. It functions as a distributed web crawler and dynamic content crawler, utilizing an XPath HTML parser to locate and extract specific data points from page structures. The framework distinguishes itself through its ability to handle dynamic content by rendering JavaScript and executing asynchronous requests to extract data from non-static pages. It also allows users to define and execute crawler logic via scripting languages, enabling the update of col

    Javacrawlerframeworkjava
    View on GitHub↗11,680
Compare all 12 related projects→