awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
kepano avatar

kepano/defuddle

0
View on GitHub↗
kepano.github.io/defuddle↗

Defuddle

Defuddle is a command line web parser and content extractor designed to isolate the primary article body from web pages and convert the result into standardized markdown. It functions as a content cleaner that removes layout clutter, such as sidebars and headers, to retrieve the main text and associated metadata.

The tool provides a terminal interface that processes content from remote URLs, local files, or piped HTML streams. It supports custom content targeting, allowing users to specify CSS selectors to manually define the main content area when automatic detection is insufficient.

The system employs heuristic-based extraction and DOM-tree sanitization to identify core content and standardize page elements. It also includes metadata schema parsing to extract structured information including titles, authors, and publication dates.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Features

  • Web Content Parsing CLI - Provides a command-line interface utility to parse web pages using URLs, local files, or piped HTML input.
  • Web Page Content Cleaning - Provides a system for isolating primary article text by removing layout clutter, sidebars, and headers from web pages.
  • Content Extractors - Extracts the main text content from webpages by removing boilerplate and navigation elements.
  • Heuristic Extraction Methods - Uses algorithms analyzing tag frequency and structural patterns to identify and extract the primary article body.
  • HTML to Markdown Converters - Transforms cleaned DOM structures into a standardized markdown format for improved readability and storage.
  • Web Article Extraction - Isolates main body text and core content from web pages to extract titles, authors, and publication dates.
  • Web Content Scrapers - Extracts information from web pages and converts the retrieved content into structured markdown.
  • CLI Input Resolvers - Implements a command-line interface that resolves input content from remote URLs, local files, or piped HTML streams.
  • Web Content Parser CLI - Provides a terminal interface for retrieving and parsing web content using URLs, local files, or piped HTML.
  • Clutter Removal - Strips non-essential HTML elements like navigation and ads to isolate core article content.
  • Body Content Extractors - Parses HTML files and retrieves the main body text by stripping markup and navigation elements.
  • Automatic Page Metadata Extraction - Retrieves structured page information including authors, titles, publication dates, and languages.
  • Selector-Based Content Targeting - Defines specific CSS selectors to extract content from websites where automatic detection fails.
  • Web Metadata Extraction - Extracts structured page information by mapping OpenGraph, JSON-LD, and meta tags to a unified data model.
  • Extraction Logic Overrides - Allows users to specify custom CSS selectors to manually define the main content area.
  • Content Extraction Overrides - Allows manual specification of CSS selectors to define main content areas when automatic detection fails.
  • Web Scraping - Processes URLs or local HTML files through a terminal to extract structured information without a browser.
3,189 stars·109 forks·TypeScript·mit·23 views

Star history

Star history chart for kepano/defuddleStar history chart for kepano/defuddle

How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

Frequently asked questions

What does kepano/defuddle do?

Defuddle is a command line web parser and content extractor designed to isolate the primary article body from web pages and convert the result into standardized markdown. It functions as a content cleaner that removes layout clutter, such as sidebars and headers, to retrieve the main text and associated metadata.

What are the main features of kepano/defuddle?

The main features of kepano/defuddle are: Web Content Parsing CLI, Web Page Content Cleaning, Content Extractors, Heuristic Extraction Methods, HTML to Markdown Converters, Web Article Extraction, Web Content Scrapers, CLI Input Resolvers.

Which projects share features with kepano/defuddle?

Projects with overlapping indexed features include: mozilla/readability — Readability is a JavaScript library designed for web content extraction. It functions as a DOM parsing utility and… postlight/parser — Postlight Parser is a command-line tool that extracts the main article content from any web page URL, returning clean… deathau/markdown-clipper — markdown-clipper is a browser extension that converts website content into markdown files for offline storage and… gsh199449/spider — Spider is a web-based platform designed for automated data extraction, providing a centralized framework to collect,… grangier/python-goose — python-goose is a Python library for web scraping and content extraction. It functions as an HTML boilerplate remover… builderio/gpt-crawler — gpt-crawler is a web scraping utility designed to extract website content and convert it into structured text files…

Projects sharing features with Defuddle

These projects share indexed features with Defuddle. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
  • mozilla/readabilitymozilla avatar

    mozilla/readability

    11,298View on GitHub↗

    Readability is a JavaScript library designed for web content extraction. It functions as a DOM parsing utility and article metadata extractor that isolates the primary text of a webpage by removing clutter such as advertisements and navigation bars. The library employs a heuristic-based content detector to predict if a webpage contains a parseable article before performing full extraction. It uses a parsing workflow to convert complex HTML documents into a simplified format, facilitating the implementation of distraction-free reader views. The tool covers several capability areas, including

    JavaScript
    View on GitHub↗11,298
  • deathau/markdown-clipperdeathau avatar

    deathau/markdown-clipper

    3,928View on GitHub↗

    markdown-clipper is a browser extension that converts website content into markdown files for offline storage and personal knowledge bases. It functions as a content extractor and HTML to markdown converter that removes layout clutter to isolate primary text. The tool includes a specific integration for sending clipped web content directly into vaults and folders within the Obsidian note-taking application. It also supports batch processing to convert all open browser tabs into individual markdown files. The extension covers a broad range of extraction capabilities, including capturing selec

    JavaScript
    View on GitHub↗3,928
  • postlight/parserpostlight avatar

    postlight/parser

    5,786View on GitHub↗

    Postlight Parser is a command-line tool that extracts the main article content from any web page URL, returning clean structured data including the title, author, date, excerpt, and lead image while stripping away ads and clutter. It uses a readability-based heuristic that scores HTML elements on text density and structural cues to identify the article body, and can accept pre-fetched HTML strings directly for parsing instead of fetching the URL. The tool distinguishes itself through a modular architecture that supports domain-specific extractor overrides, allowing custom JavaScript modules t

    JavaScriptjestlabsmercury
    View on GitHub↗5,786
  • gsh199449/spidergsh199449 avatar

    gsh199449/spider

    997View on GitHub↗

    Spider is a web-based platform designed for automated data extraction, providing a centralized framework to collect, process, and route structured information from websites. It functions as a comprehensive pipeline that manages the entire lifecycle of data gathering, from initial configuration to final storage in external databases or message queues. The platform distinguishes itself through a visual configuration interface that allows users to define extraction rules and manage scraping templates without writing custom code. It supports both static and dynamic content retrieval by integratin

    Javacralwergatherplatformspider
    View on GitHub↗997
Compare all 30 related projects→