Awesome GitHub RepositoriesContent Extraction

Odysseus is a self-hosted AI workspace and autonomous agent framework designed for deploying and managing large language models. It serves as a centralized platform for orchestrating agentic tasks, utilizing a model context protocol server to connect AI models to external system utilities, browser automation, and local hardware. The system distinguishes itself through a combination of retrieval-augmented generation and a RAG knowledge base, using vector stores and local embeddings to provide persistent semantic memory. It further integrates AI-driven communication management to triage email i

Trilium is a hierarchical personal knowledge base and digital garden tool designed to organize information into a tree of linked nodes. It functions as a JavaScript programmable wiki and a self-hosted note server, allowing users to maintain a private database of documents synchronized across multiple devices. The platform distinguishes itself through an embedded scripting engine for automating tasks and a REST API that exposes internal data and actions to external tools. Users can further extend the system by modifying the user interface layout and styling through a custom theme engine. The

Marker is an LLM-powered document parser and OCR pipeline designed to convert PDFs and unstructured files into structured markdown, JSON, and HTML. It functions as a data preprocessor that transforms complex documents into machine-readable formats while preserving tables, equations, and layout structures. The system utilizes large language models to refine OCR accuracy, clean mathematical notation, and merge fragmented tables across multiple pages. It employs model-based layout analysis to predict block types and bounding boxes, ensuring a more precise conversion of document elements. Capabi

Cheerio is an HTML and XML parsing library and server-side DOM implementation. It functions as a markup manipulation tool and CSS selector engine, allowing users to parse, query, and modify HTML or XML documents in non-browser environments. The project provides a DOM-like tree representation of markup strings, enabling programmatic addition, removal, and modification of elements and attributes. It features a prototype-based plugin system that allows the extension of core functionality by adding custom methods to the document prototype. The library covers a broad range of capabilities includi

This project is a Python framework for building autonomous, event-driven agent systems. It provides a unified runtime for orchestrating multi-agent workflows, managing persistent conversation state, and executing code within secure, isolated sandbox environments. The framework is designed to handle complex task delegation, allowing agents to invoke other agents as tools while maintaining context across multi-turn interactions. The framework distinguishes itself through its deep integration with the Model Context Protocol, enabling agents to connect to external data sources and remote services

This project is a PDF data extraction tool and document preprocessor designed to convert PDF files into structured formats such as Markdown, JSON, and HTML. It functions as an OCR document parser for scanned files, an accessibility automator for generating PDF/UA compliant metadata, and a loader for AI orchestration frameworks like LangChain. The software distinguishes itself through specialized handling of complex document elements, including the conversion of mathematical formulas into LaTeX and the generation of natural-language descriptions for charts and images. It utilizes recursive seg

Kotaemon is an orchestration framework designed for building modular, agentic workflows that integrate document processing, retrieval-augmented generation, and multi-step reasoning. It provides a comprehensive platform for developing document-based question answering systems, allowing users to chain language models, prompt templates, and external tools into complex, automated pipelines. The system distinguishes itself through a highly modular architecture that emphasizes component-based composition and schema-driven data exchange. It supports autonomous agents capable of decomposing complex q

Calibre is a comprehensive suite for digital library management, serving as a centralized hub for organizing, converting, and editing e-book collections. It functions as a multi-purpose platform that combines a relational database for metadata tracking with a powerful processing engine capable of transforming document formats and restructuring internal markup. Beyond local management, the software acts as a content server, enabling users to host their libraries over a network for remote access and reading via standard web browsers. The project distinguishes itself through its deep extensibili

Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob

Nature-skills is a suite of specialized software components designed for academic writing assistance, literature retrieval, document conversion, peer review simulation, and scientific figure generation. It functions as an LLM-driven assistant to help researchers polish scientific manuscripts, manage bibliographic references, and produce publication-quality materials for high-impact journals. The project distinguishes itself through a multi-stage writing pipeline and an agentic literature retrieval system that verifies citations across academic databases. It includes capabilities for convertin

gpt-crawler is a web scraping utility designed to extract website content and convert it into structured text files for use as AI model knowledge bases. It functions as a data generator that crawls specified web addresses to produce the knowledge files required for building custom GPTs, grounding large language models, and providing context to AI agents. The system transforms raw HTML into clean Markdown text to reduce token usage and improve readability for AI models. It utilizes token-aware content chunking and output file size limitations to ensure generated datasets remain compatible with

Handsontable is a JavaScript data grid that provides a spreadsheet-like interface for managing and editing large datasets within web applications. It functions as a virtualized data table that renders only visible cells to maintain performance, paired with a synchronization layer that binds the grid to underlying data structures. The project distinguishes itself through a built-in spreadsheet calculation engine for evaluating mathematical and logical expressions and a dedicated tool for exporting grid content into Excel XLSX files. It ensures interoperability with external spreadsheet softwar

This project is an automation suite comprising an AI visual asset generator, a browser-based social publisher, an Electron resource extractor, and a Markdown content transformer. It functions as a content automation pipeline that uses large language models to generate text and images for distribution across social media platforms. The system distinguishes itself through specialized visual generation capabilities, producing professional infographics, slide decks, educational comics, and SVG diagrams via structured prompts. It also features a dedicated workflow for extracting resources from Ele

FreeTube is a privacy-focused desktop application for watching YouTube videos without ads, tracking cookies, or the requirement of a Google account. It functions as a local-first subscription manager that tracks channels and playlists in local files instead of a centralized cloud account. The application avoids tracking-heavy official APIs by using a content extractor that parses web pages directly. To further protect user identity, it can route network traffic through proxies or Tor to mask the hardware IP address. The software provides tools for distraction-free viewing, including the abil

Letta is a framework for building, deploying, and managing autonomous AI agents that maintain persistent state across long-term interactions. It provides a comprehensive suite of primitives for defining agents with configurable personas, modular memory blocks, and tool-use capabilities, enabling them to retain user preferences and conversation history over extended sessions. The platform distinguishes itself through its advanced memory management and orchestration capabilities. It allows agents to autonomously update their own memory, perform retrieval-augmented generation, and coordinate com

This project is a comprehensive framework for building and managing autonomous agent systems. It provides a unified architecture for orchestrating multi-agent societies, where specialized agents collaborate through roleplay to decompose and solve complex tasks. The system integrates language models with external environments, enabling agents to perform real-world actions through a standardized tool-calling abstraction layer. The framework distinguishes itself through its focus on iterative reasoning and data reliability. It employs automated feedback loops to refine agent outputs and self-eva

Newspaper is a Python library designed for scraping, parsing, and analyzing web-based information. It functions as a framework for automated news aggregation and large-scale web content extraction, providing tools to download, clean, and structure text, metadata, and media from diverse online sources. The project distinguishes itself through a pipeline-oriented architecture that combines heuristic-based content extraction with natural language processing. It automatically identifies and isolates article bodies from web page boilerplate while simultaneously performing language detection, keywo

This project is a collection of Python scripts and tools designed for web scraping, browser automation, and large-scale data extraction. It provides a set of implementations for retrieving information from websites and private APIs, including tools for multimedia downloading and social media data archiving. The toolset includes specialized mechanisms for bypassing anti-scraping measures through IP proxy pool rotation and multi-threaded crawlers. It also features capabilities for simulating browser sessions to handle authentication, intercepting session cookies, and decrypting network payloads

FreshRSS is an open-source, self-hosted web feed aggregator designed to collect, organize, and display content from multiple websites in a single, centralized interface. It functions as a comprehensive reader for standard syndication formats, allowing users to track updates from various sources while maintaining full control over their data and privacy. The platform supports multi-user environments, enabling individual account management and personalized reading experiences. The application distinguishes itself through its robust synchronization and extensibility capabilities. It provides a s

Notesnook is a cross-platform note-taking application designed for private knowledge management. It functions as a secure personal information manager that prioritizes user data confidentiality by employing client-side end-to-end encryption for all stored notes, attachments, and metadata. By utilizing a local-first synchronization model, the application ensures that information remains accessible offline while maintaining a consistent state across desktop and mobile devices. The platform distinguishes itself through a focus on privacy-centric organization and secure information handling. User