Open-source alternatives to Defuddle

Spider is a web-based platform designed for automated data extraction, providing a centralized framework to collect, process, and route structured information from websites. It functions as a comprehensive pipeline that manages the entire lifecycle of data gathering, from initial configuration to final storage in external databases or message queues. The platform distinguishes itself through a visual configuration interface that allows users to define extraction rules and manage scraping templates without writing custom code. It supports both static and dynamic content retrieval by integratin

python-goose is a Python library for web scraping and content extraction. It functions as an HTML boilerplate remover and article parser designed to isolate primary text and metadata from web pages by stripping away navigation, layout noise, and non-essential elements. The tool features multilingual processing capabilities, utilizing language-specific stop-word analyzers to identify and extract primary content across different languages. It also identifies and collects embedded media, including source URLs and embed codes for lead images and videos associated with an article. The library cov

Wechatsync is a multi-platform content synchronizer and cross-platform publishing tool. It extracts articles from webpages and distributes them to multiple social media and blogging platforms simultaneously. The system utilizes a web content extractor with reader-mode logic to strip advertisements and navigation elements from source pages. The project employs a markdown content pipeline that converts extracted web content into a standardized format for editing before redistribution. It features an automated media migrator that performs host-to-host image migration, downloading images from sou

gpt-crawler is a web scraping utility designed to extract website content and convert it into structured text files for use as AI model knowledge bases. It functions as a data generator that crawls specified web addresses to produce the knowledge files required for building custom GPTs, grounding large language models, and providing context to AI agents. The system transforms raw HTML into clean Markdown text to reduce token usage and improve readability for AI models. It utilizes token-aware content chunking and output file size limitations to ensure generated datasets remain compatible with

Goutte is a PHP web scraper and DOM crawler designed for extracting data from websites. It functions as an HTTP client wrapper that enables the retrieval of web pages and the parsing of HTML content. The project provides a web form automator to programmatically fill and submit HTML forms to remote servers. It also includes a mechanism for automated website crawling by following links to discover and archive web content. The system supports stateful session management to maintain cookies and headers across requests. It further covers HTML data extraction through DOM-based element selection an

Trafilatura is a Python library and command-line tool for extracting clean, structured text and metadata from web pages. It downloads HTML content, identifies the main body of text, and strips away navigation, ads, and other boilerplate, returning the core article content along with fields like title, author, date, and URL. The tool can also extract user comments and test whether a page contains extractable text, making it a general-purpose web text extraction library. What distinguishes Trafilatura from simpler extractors is its configurable extraction pipeline, which offers high-speed, high

This project is a comprehensive resource directory for web data extraction, providing a curated collection of tools and libraries for parsing data, automating browsers, and managing network operations. It serves as a guide for extracting structured information from HTML, XML, JSON, and PDF formats. The toolkit focuses on advanced data collection strategies, including headless browser automation to interact with JavaScript and a suite of network utilities for DNS resolution and WebSocket connections. It specifically covers methods for bypassing bot protections through proxy pool management, us

iflow-cli is a command-line interface and suite of AI tools designed for software engineering, workflow orchestration, and multimodal data analysis. It functions as an LLM command line interface that enables users to execute AI workflows, analyze codebase structures, and interact with large language models directly from the terminal. The project features a plugin-based agent architecture that allows for the integration of specialized domain experts and custom instruction sets from an external marketplace. It distinguishes itself through a multimodal AI terminal capable of processing visual da

Firecrawl is a web data extraction platform designed to convert unstructured web content into clean, LLM-ready formats like markdown or JSON. It functions as an autonomous web crawler and scraper, capable of mapping entire domains, performing recursive navigation, and executing complex data gathering tasks. By leveraging headless browser orchestration, the system handles dynamic, JavaScript-heavy pages to ensure comprehensive data capture. The platform distinguishes itself through its focus on agentic workflows, providing a programmatic interface that allows autonomous agents to perform live

Langroid is a multi-agent orchestration framework and tool integration suite designed for building complex AI applications. It serves as a multi-modal integration layer that connects diverse local and remote language models with an agentic retrieval-augmented generation system. The project distinguishes itself through a collaborative message-exchange paradigm, allowing specialized agents to delegate tasks hierarchically and coordinate via structured communication. It features an advanced state management system for conversational AI, including the ability to rewind and prune conversation hist

GeneralNewsExtractor is a specialized system for identifying and extracting structured news data through configurable HTML filtering and XPath-based parsing. It functions as an automated news scraper and web page content parser that isolates article bodies and metadata from news webpages. The system uses YAML or JSON configuration files to manage extraction rules and XPath definitions, decoupling the processing logic from site-specific selectors. This allows for targeted noise removal and the isolation of titles, authors, and publication dates through customizable filtering. The tool covers

Min is a minimalist, privacy-focused web browser designed to limit data collection and remove interface clutter. It serves as an ad-blocking tool that prevents tracking scripts and advertisements from loading to improve page speed and protect user identity. The browser differentiates itself through organized tab management, where related tabs are grouped into named tasks to separate work streams. It features a tag-based bookmark manager that replaces traditional hierarchical folders with custom labels and provides a simplified reader view to strip away non-essential page elements for focused

html-to-markdown is a tool that transforms HTML content and websites into Markdown text. It functions as an HTML to Markdown converter and a command line processor for converting files or standard input streams into Markdown documents. The project is an extensible conversion engine that allows for custom element rendering and the registration of plugins to modify the transformation process. It includes a content filter that uses CSS selectors to include or exclude specific HTML elements during conversion. The system covers broad capability areas including link resolution for relative URLs, r

This project is a digital library exporter and EPUB content generator specifically designed to download and convert books from the O'Reilly Learning platform into electronic publication files for offline reading and permanent storage. It functions as a web content scraper that retrieves restricted text and media to facilitate digital library archiving. The tool manages session authentication using credentials or cookies to maintain continuous access to content. It retrieves books via unique identifiers and transforms the source material into standardized EPUB files, including layout optimizat

This project is a Model Context Protocol server that connects large language models to web scraping and crawling tools. It functions as a bridge, allowing LLM clients to utilize a web crawling engine and scraping utilities to extract and process web data. The server integrates a markdown web converter that transforms dynamic web pages and PDF documents into clean markdown to optimize consumption by AI models. It also provides a browser automation interface for controlling headless sessions and bypassing access restrictions. The system covers broad capabilities including large-scale website d

Summarize is a command line tool and multimodal content extractor designed to generate concise summaries from web pages, documents, and media files. It functions as an orchestrator that connects developer tools to various language model providers to process and condense information. The system provides specialized capabilities for audio and video processing, including transcription with speaker identification and the extraction of timestamped visual markers from video slides. It also includes a translation utility to convert generated summaries and extracted text into different target languag

AnyCrawl is an AI-powered data extractor, automated web crawler, and headless browser orchestrator. It serves as a web content extraction API and a gateway that connects crawling and scraping tools to language models using a standardized API protocol. The project specializes in converting unstructured website content into structured JSON or markdown optimized for AI assistants. It utilizes language models and JSON schemas to pull specific information into validated formats and provides capabilities for AI page summarization and LLM-optimized content extraction. The system manages comprehensi

Markdownload is a browser extension that functions as a markdown web clipper, converting webpages and selected text into clean markdown files for offline storage and archiving. It operates as a content extractor that isolates the main document from the page while removing navigation elements and advertisements. The tool includes a template generator for injecting dynamic front-matter and metadata into documents via user-defined placeholders. It also serves as a local media downloader that saves remote images to the filesystem and updates links to reference those local files. Additionally, it

Vditor is a browser-based Markdown editor and rendering engine that supports multiple editing interfaces, including a visual rich-text experience, instant rendering, and a traditional side-by-side split-view preview. It serves as an authoring tool for technical documentation and a component for web-based editor integration. The project is distinguished by its support for complex technical content, utilizing specialized rendering for mathematical formulas, flowcharts, sequence diagrams, and mind maps. It also functions as a collaborative document review tool, enabling users to attach anchored

Percollate is a command-line tool for converting web pages and RSS feeds into structured files. It functions as a web content converter, static document generator, and page bundler that transforms online content into PDF, EPUB, HTML, or Markdown formats. The tool creates self-contained documents by embedding external images as encoded data URLs and applying custom HTML templates and CSS stylesheets. It can combine multiple web URLs or feed entries into a single digital book featuring a generated table of contents and hyperlinked index. Additional capabilities include the decomposition of Ato

Open Deep Research is an AI-powered web research agent that combines a reasoning model with live web search and data extraction to perform deep, multi-source investigations on any topic. It operates through a dual interface, offering both a command-line tool and a Model Context Protocol server, allowing developers to integrate web capabilities directly into AI agents and coding assistants. The project distinguishes itself by orchestrating an iterative research loop where a reasoning model plans steps, interprets search results, and guides subsequent web interactions. It uses Firecrawl for scr

Shiori is a self-hosted bookmark manager and webpage archiving tool. Written in Go, it functions as a backend service that allows users to save, organize, and search for web links while maintaining a private collection of online resources. The system ensures content availability by creating offline copies of saved pages, preventing data loss if the original source is removed. It is distributed as a containerized application to provide consistent installation and deployment across different operating systems. The software provides a dual-interface access model, featuring both a web-based mana

wewe-rss is an RSS feed generator and scheduled aggregator that converts social media accounts and web content into standardized RSS, Atom, or JSON feeds. It functions as a full-text content extractor, retrieving the complete body of articles rather than short summaries. The system operates as an API-protected feed gateway, utilizing token-based authorization and request rate limiting to restrict access and maintain stability. It supports subscription portability by exporting tracked sources into standardized OPML files. The project manages content aggregation through automated polling via c

Omniparse is a multimodal content parser and generative AI ingestion engine designed to convert documents, images, and multimedia into a uniform format. It functions as a data preprocessing pipeline that transforms diverse raw data sources into structured markdown to improve the performance of large language model workflows. The system extracts text and structural data from PDFs, images, audio, and video files. It includes a web crawler that converts dynamic website content into clean markdown and a multimodal transformation process that maps disparate input formats into a unified data schema

Reader is an AI data ingestion pipeline and web content parser designed to convert websites and documents into clean markdown for use with large language models. It functions as a headless browser content extractor and web-to-markdown converter, transforming URLs and PDF files into structured text formats while removing irrelevant web clutter. The system optimizes retrieval augmented generation by acting as a search optimizer that retrieves web results and applies re-ranking to improve context relevance. It further enhances content accessibility by using vision models to generate descriptive

ReadYou is a self-hosted reading application and RSS feed aggregator that centralizes content from multiple web sources. It functions as a full-text RSS reader, extracting the complete body text from web pages to provide a distraction-free reading experience. The application includes specialized accessibility and speed tools, such as a bionic reading mode that uses pattern-based text highlighting to guide the eye and a text-to-speech system for audio content consumption. The project covers comprehensive subscription management through OPML import and export, feed categorization, and keyword-

Web Clipper is a browser extension that captures web content and saves it directly to a variety of note-taking and productivity platforms. It strips distracting page elements like ads and sidebars before clipping, ensuring only the core article, recipe, or product information is stored cleanly. The extension supports saving content to multiple destinations including Notion, OneNote, Obsidian, Joplin, Confluence, and GitHub, allowing users to send web clippings to their preferred workspace with a single action. It uses a plugin-based architecture where each platform is wrapped in an adapter th