adbar/trafilatura
Trafilatura
Trafilatura is a Python library and command-line tool for extracting clean, structured text and metadata from web pages. It downloads HTML content, identifies the main body of text, and strips away navigation, ads, and other boilerplate, returning the core article content along with fields like title, author, date, and URL. The tool can also extract user comments and test whether a page contains extractable text, making it a general-purpose web text extraction library.
What distinguishes Trafilatura from simpler extractors is its configurable extraction pipeline, which offers high-speed, high-precision, or exhaustive modes to balance quality and performance. It supports parallel URL fetching with a configurable worker pool, discovers page URLs by parsing XML sitemaps and RSS/Atom feeds, and can fall back to the Internet Archive when a page is unreachable. An optional language detection module identifies and filters extracted text by language code, and the output can be serialized into plain text, Markdown, CSV, JSON, HTML, XML, and XML-TEI formats for integration into downstream tools.
Beyond single-page extraction, Trafilatura handles batch workflows: it can crawl entire websites via internal link following, process directories of local HTML files, and bulk download a list of URLs. Element-level filtering allows users to keep or discard tables, images, links, and comments during extraction. The tool is available as a Python package and provides a straightforward command-line interface alongside its library API.
Features
- Full-Text Content Extraction - Extracts the main body text from web pages while stripping navigation, ads, and boilerplate.
- Extraction Element Filters - Selectively retains or discards HTML elements such as tables, comments, links, and images during extraction based on user flags.
- Web Crawling - Discovers pages by following internal links from a starting URL with configurable depth.
- Extraction Mode Selectors - Switches between high-speed, high-precision, and exhaustive extraction modes by adjusting internal heuristics and element filters.
- Heuristic Extraction Methods - Identifies main article text by analysing tag frequency, text density, and structural heuristics within the parsed HTML tree.
- Metadata Extraction - Trafilatura extracts metadata fields such as date, title, author, and URL from HTML pages with configurable parameters.
- Web Scrapers - A command-line tool that extracts text from URLs or HTML files, with options for recursive crawling from sitemaps and feeds.
- PDF and HTML Content Extraction - Provides a library API for downloading and extracting clean text and metadata from HTML pages.
- Web Page Retrievers - Fetches raw HTML from URLs, handling encoding and security.
- Bulk URL Processing - Processes multiple URLs from a file to download and extract text sequentially.
- Automatic Page Metadata Extraction - Extracts structured metadata like author, title, and date from HTML pages.
- Content Formats and Exporting - Converts extracted web content to plain text, Markdown, CSV, JSON, HTML, XML, and TEI.
- Multi-Format Output Converters - Converts extracted web content into plain text, Markdown, CSV, JSON, HTML, XML, and XML-TEI for integration into other tools.
- Batch Input Processing - Trafilatura reads all HTML files from an input directory and writes extracted results to an output directory.
- Output Format Rendering - Supports exporting extracted content to text, Markdown, CSV, JSON, HTML, XML, and TEI.
- URL Batch Processors - Fetches multiple URLs concurrently with deduplication and archive fallback.
- Content and Language Filtering - Detects extracted text language and filters results to pages matching a specified language code.
- Command Line Interfaces - Provides a command-line interface for direct extraction from URLs and files.
- Concurrent Request Pooling - Manages concurrent network requests with a configurable worker pool, deduplicating URLs and handling rate limits.
- Sitemap Crawlers - Discovers page URLs by parsing sitemaps and RSS/Atom feeds for batch extraction.
- Feed - Discovers pages by parsing sitemaps and feeds then extracts their content in batch.
- Feed - Processes sitemaps and web feeds to discover page URLs for extraction.
- Data Scraping Tools - Python tool for gathering text and metadata from web pages.
- Web Scraping - Extracts clean text from web pages for training corpora.
- Web Scraping - Tool to gather text and metadata from the web.
