Defuddle

Features

Web Content Parsing CLI - Provides a command-line interface utility to parse web pages using URLs, local files, or piped HTML input.

Web Page Content Cleaning - Provides a system for isolating primary article text by removing layout clutter, sidebars, and headers from web pages.

Content Extractors - Extracts the main text content from webpages by removing boilerplate and navigation elements.

Heuristic Extraction Methods - Uses algorithms analyzing tag frequency and structural patterns to identify and extract the primary article body.

HTML to Markdown Converters - Transforms cleaned DOM structures into a standardized markdown format for improved readability and storage.

Web Article Extraction - Isolates main body text and core content from web pages to extract titles, authors, and publication dates.

Web Content Scrapers - Extracts information from web pages and converts the retrieved content into structured markdown.

CLI Input Resolvers - Implements a command-line interface that resolves input content from remote URLs, local files, or piped HTML streams.

Web Content Parser CLI - Provides a terminal interface for retrieving and parsing web content using URLs, local files, or piped HTML.

Clutter Removal - Strips non-essential HTML elements like navigation and ads to isolate core article content.

Body Content Extractors - Parses HTML files and retrieves the main body text by stripping markup and navigation elements.

Automatic Page Metadata Extraction - Retrieves structured page information including authors, titles, publication dates, and languages.

Selector-Based Content Targeting - Defines specific CSS selectors to extract content from websites where automatic detection fails.

Web Metadata Extraction - Extracts structured page information by mapping OpenGraph, JSON-LD, and meta tags to a unified data model.

Extraction Logic Overrides - Allows users to specify custom CSS selectors to manually define the main content area.

Content Extraction Overrides - Allows manual specification of CSS selectors to define main content areas when automatic detection fails.

Web Scraping - Processes URLs or local HTML files through a terminal to extract structured information without a browser.

Defuddle is a command line web parser and content extractor designed to isolate the primary article body from web pages and convert the result into standardized markdown. It functions as a content cleaner that removes layout clutter, such as sidebars and headers, to retrieve the main text and associated metadata.

The tool provides a terminal interface that processes content from remote URLs, local files, or piped HTML streams. It supports custom content targeting, allowing users to specify CSS selectors to manually define the main content area when automatic detection is insufficient.

The system employs heuristic-based extraction and DOM-tree sanitization to identify core content and standardize page elements. It also includes metadata schema parsing to extract structured information including titles, authors, and publication dates.

Features