lycheeverse/lychee
Lychee
Lychee is a command line link checker written in Rust that scans websites and local files to identify broken hyperlinks and invalid email addresses. It functions as an asynchronous link validator and static site auditor designed to ensure content accessibility and quality.
The tool features disk-based response caching to accelerate subsequent scans and avoid redundant network requests. It supports token-based API authentication for services like GitHub to bypass rate limits when verifying links on remote repositories.
The project covers a range of capabilities including multi-format content parsing for HTML and Markdown, relative path resolution, and regex-based link filtering. It can be integrated into continuous integration pipelines and pre-commit hooks to automate link validation during the development process.
Features
- Broken Link Scanners - Scans websites and local files to identify dead hyperlinks and invalid email addresses.
- Link Integrity Verifiers - Verifies the validity of hyperlinks and email addresses using configurable HTTP methods and timeouts.
- Static Site Link Auditors - Detects dead links in local directories or remote sites to ensure content accessibility and quality.
- Asynchronous Validators - Implements a stream-based validator that checks URL integrity across formats using asynchronous requests.
- Link Extraction - Identifies URLs within HTML, Markdown, and plain text files using diverse parsing methods.
- Continuous Integration Validators - Provides an automated engine for checking hyperlink integrity within CI pipelines and pre-commit hooks.
- Link Extraction - Extracts URLs from HTML, Markdown, and plain text files using specialized parsing logic for each format.
- CI/CD Pipeline Integrations - Integrates into continuous integration pipelines and pre-commit hooks to automate link validation.
- Continuous Integration Checks - Integrates into CI/CD pipelines and pre-commit hooks to automate broken link detection.
- Hyperlink Management - Scans text files and websites to identify broken hyperlinks and invalid email addresses.
- URL Relative Path Resolvers - Resolves relative hyperlinks into absolute addresses based on a root directory or base URL.
- Link Checkers - A command line tool written in Rust that scans websites and files for broken hyperlinks.
- Multi-Format Extractors - Extracts URLs from HTML and Markdown files while using plain text extraction for other supported formats.
- Quality Assurance Practices - Verifies that all links in Markdown and HTML files are functional to ensure documentation quality.
- Web Crawling and Scraping - Crawls remote websites to extract and verify the reachability of all hyperlinks.
- Concurrent Stream Processing - Implements concurrent processing of network requests and file reads via asynchronous streams to maximize validation throughput.
- Analysis Result Caches - Caches previous link validation outcomes on disk to avoid redundant network requests across runs.
- HTTP Response Caching - Caches HTTP responses on disk to avoid redundant network requests across multiple scanning runs.
- Runtime Request Caching - Saves HTTP responses to disk to accelerate subsequent validation runs and minimize network traffic.
- Email Address Validators - Verifies that email addresses found in text or websites are valid and reachable.
- GitHub Authentication - Integrates GitHub personal access tokens to bypass API rate limits when checking remote repository links.
- URL Regex Filters - Provides regular expression filters to exclude specific URLs or directories from the scanning process.
- API Request Authentication - Authenticates requests to remote repositories using security tokens to ensure continuous link verification.
- Token-Based Authentication - Uses security tokens in HTTP headers to authenticate requests and bypass API rate limits.
- High-Throughput Link Validation - Checks thousands of links efficiently using async Rust streams and local caching.
- Quality Assurance Tools - Fast link checker for documentation and websites.
