1 repositorio
Awesome GitHub RepositoriesFetch Control Policies
Configuration options for controlling how content is retrieved, including redirect following and encryption handling.
Distinct from Web Content Fetching: Distinct from Web Content Fetching: focuses on the behavioral policies of the fetcher (redirects, SSL) rather than the format conversion for LLMs.
Explore 1 awesome GitHub repository matching data & databases · Fetch Control Policies. Refine with filters or upvote what's useful.
Awesome Fetch Control Policies GitHub Repositories
Encuentra los mejores repositorios con IA.Buscaremos los repositorios que mejor coincidan usando IA.
- yasserg/crawler4j
yasserg/crawler4j
4,622Ver en GitHubCrawler4j is a multi-threaded Java web crawler and spider designed for high-volume web traversal and content extraction. It functions as a polite crawling framework that enables the discovery and indexing of HTML and binary content across multiple websites. The project distinguishes itself through a persistent crawling model that serializes session state to local storage, allowing the engine to resume indexing after a crash or interruption. It includes a politeness controller to regulate request frequency and delays, preventing server overloading and IP blocking. The system covers a broad ra
Provides control over whether to follow redirects, include encrypted pages, or process specific content types.
Java
