yasserg/crawler4j
Crawler4j
Crawler4j is a multi-threaded Java web crawler and spider designed for high-volume web traversal and content extraction. It functions as a polite crawling framework that enables the discovery and indexing of HTML and binary content across multiple websites.
The project distinguishes itself through a persistent crawling model that serializes session state to local storage, allowing the engine to resume indexing after a crash or interruption. It includes a politeness controller to regulate request frequency and delays, preventing server overloading and IP blocking.
The system covers a broad range of traversal capabilities, including depth-limited scope management, target filtering, and request interception for custom user agents and proxy routing. Data storage is handled via a repository pattern that decouples crawling logic from the persistence of page metadata in relational databases.
Features
- Web Traversal Engines - Ships a multi-threaded engine for high-throughput web traversal and content extraction.
- Multi-Threaded Web Traversal - Uses a multi-threaded engine to concurrently fetch and process multiple web pages for high-throughput indexing.
- Crawl Progress Persisters - Serializes crawl progress and frontier data to local storage to enable session recovery after interruptions.
- Web Spiders - Acts as a high-volume discovery tool that follows links to fetch HTML and binary content.
- Crawl Target Filtering - Implements custom logic to determine which URLs to visit based on domain restrictions and file type exclusions.
- Web Crawlers - Provides a persistent crawler that saves session state to resume indexing after interruptions.
- Web Data Extraction - Automates the fetching of HTML and binary files to extract structured data for storage.
- Crawl Depth Limiters - Includes a depth-limiting mechanism to control the scale of web traversal and manage resources.
- Crawl Scope Management - Provides mechanisms to set limits on crawl depth and the total number of pages fetched to manage resources.
- Crawl Politeness Controllers - Implements a politeness controller to regulate request frequency and prevent server overloading.
- Crawl State Recovery - Persists the crawl state to local folders to allow resuming from a specific point after a crash.
- Crawling Request Throttlers - Implements a politeness controller that regulates request frequency and adds delays to prevent server overloading.
- Web Crawling - Provides a multi-threaded system designed for large-scale discovery and indexing of web content.
- Web Content Harvesting - Provides a framework for automated web harvesting while managing request rates to avoid overloading servers.
- Web Crawlers - Implements a multi-threaded web crawler engine using the Java programming language.
- Web Crawling Frameworks - Functions as a polite crawling framework that regulates request rates to avoid server overloading.
- Relational Database Storage - Supports persisting captured page content and metadata into relational databases.
- Fetch Control Policies - Provides control over whether to follow redirects, include encrypted pages, or process specific content types.
- Search & Indexing Engines - Traverses the web to collect pages for a searchable directory or index using custom filters.
- Web Scrapers - Allows configuration of crawl depth, user agents, and proxy routing to control data collection scope.
- Request Interceptors - Provides a request interceptor pipeline to inject custom user agents and route traffic through proxies.
- Binary Asset Crawling - Enables the discovery and downloading of non-textual assets such as images and audio files.
- Java Crawling Frameworks - Simple and lightweight web crawler.
- Web Crawling - Simple and lightweight web crawler.
