yasserg/crawler4j
Crawler4j
Crawler4j is a multi-threaded Java web crawler and spider designed for high-volume web traversal and content extraction. It functions as a polite crawling framework that enables the discovery and indexing of HTML and binary content across multiple websites.
The project distinguishes itself through a persistent crawling model that serializes session state to local storage, allowing the engine to resume indexing after a crash or interruption. It includes a politeness controller to regulate request frequency and delays, preventing server overloading and IP blocking.
The system covers a broad range of traversal capabilities, including depth-limited scope management, target filtering, and request interception for custom user agents and proxy routing. Data storage is handled via a repository pattern that decouples crawling logic from the persistence of page metadata in relational databases.
Features
- Web Traversal Engines - Ships a multi-threaded engine for high-throughput web traversal and content extraction.
- Multi-Threaded Web Traversal - Uses a multi-threaded engine to concurrently fetch and process multiple web pages for high-throughput indexing.
- Crawl Progress Persisters - Serializes crawl progress and frontier data to local storage to enable session recovery after interruptions.
- Web Spiders - Acts as a high-volume discovery tool that follows links to fetch HTML and binary content.
- Crawl Target Filtering - Implements custom logic to determine which URLs to visit based on domain restrictions and file type exclusions.
- Web Crawlers - Provides a persistent crawler that saves session state to resume indexing after interruptions.
- Web Data Extraction - Automates the fetching of HTML and binary files to extract structured data for storage.
- Crawl Depth Limiters - Includes a depth-limiting mechanism to control the scale of web traversal and manage resources.
- Crawl Scope Management - Provides mechanisms to set limits on crawl depth and the total number of pages fetched to manage resources.
- Crawl Politeness Controllers - Implements a politeness controller to regulate request frequency and prevent server overloading.
- Crawl State Recovery - Persists the crawl state to local folders to allow resuming from a specific point after a crash.
- Crawling Request Throttlers - Implements a politeness controller that regulates request frequency and adds delays to prevent server overloading.
- Web Crawling - Provides a multi-threaded system designed for large-scale discovery and indexing of web content.
- Web Content Harvesting - Provides a framework for automated web harvesting while managing request rates to avoid overloading servers.
- Web Crawlers - Implements a multi-threaded web crawler engine using the Java programming language.
- Web Crawling Frameworks - Functions as a polite crawling framework that regulates request rates to avoid server overloading.
- Relational Database Storage - Supports persisting captured page content and metadata into relational databases.
- Fetch Control Policies - Provides control over whether to follow redirects, include encrypted pages, or process specific content types.
- Search & Indexing Engines - Traverses the web to collect pages for a searchable directory or index using custom filters.
- Web Scrapers - Allows configuration of crawl depth, user agents, and proxy routing to control data collection scope.
- Request Interceptors - Provides a request interceptor pipeline to inject custom user agents and route traffic through proxies.
- Binary Asset Crawling - Enables the discovery and downloading of non-textual assets such as images and audio files.
- Java Crawling Frameworks - Simple and lightweight web crawler.
- Web Crawling - Simple and lightweight web crawler.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
