Awesome GitHub RepositoriesExtraction Configurations

Docling is a modular framework designed for document parsing, layout analysis, and structured data extraction. It transforms unstructured files and web content into a unified, hierarchical data model that preserves the spatial and semantic relationships between text, tables, images, and layout elements. By normalizing diverse input formats into a consistent internal representation, the library enables uniform processing across various document types. The project distinguishes itself through a schema-driven approach that maps document regions to strongly-typed objects, ensuring data accuracy t

Newspaper is a Python library designed for scraping, parsing, and analyzing web-based information. It functions as a framework for automated news aggregation and large-scale web content extraction, providing tools to download, clean, and structure text, metadata, and media from diverse online sources. The project distinguishes itself through a pipeline-oriented architecture that combines heuristic-based content extraction with natural language processing. It automatically identifies and isolates article bodies from web page boilerplate while simultaneously performing language detection, keywo

Instructor is a framework designed for structured data extraction, validation, and language model integration. It functions as a library that transforms unstructured text into validated, type-safe objects by leveraging schema definitions and model-specific tool-calling capabilities. By acting as a validation middleware, the project ensures that language model outputs strictly conform to defined data structures. The library distinguishes itself through a robust validation-based retry loop that automatically re-submits failed responses with error feedback to iteratively correct schema complianc

Kreuzberg is a document extraction engine that converts PDFs, Office files, images, and over 90 other formats into clean, structured text and metadata. It is built around a compiled Rust core that can be used as a native library, a command-line tool, a REST API server, or a WebAssembly module for browser-based processing. The system is designed to run entirely on self-hosted infrastructure, with no data leaving the user's environment. What distinguishes Kreuzberg is its breadth of integration surfaces and its pipeline architecture. It exposes extraction capabilities through native bindings fo

Proxyman is a cross-platform HTTP debugging proxy that captures, inspects, and modifies HTTP, HTTPS, and WebSocket traffic. It functions as a man-in-the-middle proxy, decrypting SSL/TLS traffic to allow real-time inspection and modification of encrypted requests and responses. The tool is designed for debugging web and mobile applications, with capabilities for API mocking and simulation, scriptable traffic modification, and team collaboration on network logs. What distinguishes Proxyman is its deep integration with mobile and cross-platform development workflows. It provides automated certif

frpc-desktop is a cross-platform desktop application that provides a graphical interface for managing FRP intranet penetration tunnels. It combines FRP binary downloading, visual configuration editing, tunnel lifecycle management, and shareable link generation into a single tool, allowing users to tunnel local network services to the public internet through a relay server without manually editing configuration files or managing binaries. The application wraps a web frontend inside an Electron shell for native window management across operating systems, and handles the full lifecycle of the FR

Data-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets to prepare them for training large language and vision models. It functions as a distributed data pipeline engine that runs processing jobs across Ray clusters, handling billions of samples with automatic operator fusion and adaptive parallelism. The framework provides a library of operators that leverage large language models for semantic extraction, filtering, and data synthesis within processing pipelines. The project distinguishes itself through a YAML-based data recipe sys

PaddleX is a PaddlePaddle-based framework for building, deploying, and fine-tuning AI model pipelines, with pre-built support for computer vision, OCR, document analysis, and time series tasks. It offers a toolkit of ready-to-use pipelines for image classification, object detection, segmentation, and pose estimation, alongside an end-to-end OCR document analysis pipeline that extracts text, tables, formulas, and layout information. The platform also includes a dedicated time series forecasting pipeline for analyzing historical data to detect anomalies, classify patterns, and predict future val

ZenML is an extensible machine learning orchestration framework designed to manage the end-to-end lifecycle of data pipelines and AI agent workflows. It functions as a durable orchestrator that executes machine learning tasks as directed acyclic graphs, ensuring that every step is containerized for consistent performance across local, cloud, and hybrid infrastructure. By decoupling pipeline code from underlying compute and storage backends, the platform allows developers to define infrastructure-agnostic stacks that remain portable across diverse environments. The project distinguishes itself

ZenML is an orchestration platform designed for building, deploying, and monitoring reproducible machine learning pipelines and agentic workflows. It provides a unified framework that manages the entire lifecycle of machine learning assets, from data processing and model training to the deployment of persistent inference services. By decoupling pipeline logic from underlying compute and storage, the platform enables teams to transition workflows seamlessly from local development environments to production-grade cloud infrastructure. The platform distinguishes itself through a service-oriented

sChandra is a document processing platform that converts images, PDFs, Word documents, spreadsheets, and other formats into structured output such as HTML, Markdown, or JSON while preserving layout. It can also extract specific data fields from invoices, contracts, or reports using user-defined JSON schemas, with citations back to source locations. The service supports form filling in PDF and image documents, document generation from Markdown, and extraction of tracked changes from Word files. The platform distinguishes itself with pipeline-based processing chains that combine multiple proces

Expose is a secure tunneling tool that creates public URLs for local development servers, making them accessible from the internet without complex network configuration. It forwards incoming HTTP requests and responses between a public relay server and a local machine, enabling external services to reach a development environment that would otherwise be isolated behind a firewall or VPN. The project distinguishes itself through a comprehensive set of webhook debugging and inspection capabilities. It captures all incoming webhook requests and displays them in a real-time dashboard or command-l

GPAC is an open-source multimedia framework built around a pluggable filter graph pipeline, where modular processing units called filters connect into a directed graph to handle media workflows. At its core, the framework centers all media packaging and manipulation on the ISO Base Media File Format (ISOBMFF), with specialized tools for reading, writing, fragmenting, and encrypting MP4 and related containers. It also provides a declarative scene graph composition system for describing interactive multimedia scenes using MPEG-4 BIFS, X3D, SVG, or VRML syntax, alongside a hardware-accelerated re

सिंथेटिक डेटा किट भाषा मॉडल के लिए प्रशिक्षण डेटासेट उत्पन्न करने, क्यूरेट करने और स्वरूपित करने के लिए डिज़ाइन किया गया एक एकीकृत फ्रेमवर्क है। यह एक एंड-टू-एंड पाइपलाइन प्रदान करता है जो कच्चे स्रोत दस्तावेज़ों को फाइन-ट्यूनिंग, तर्क और टूल-उपयोग मॉडल प्रशिक्षण के लिए उपयुक्त संरचित डेटा में परिवर्तित करता है। यह फ्रेमवर्क एक मॉड्यूलर ऑर्केस्ट्रेशन इंजन के माध्यम से खुद को अलग करता है जो डेटा तैयारी के पूरे लाइफसाइकिल का प्रबंधन करता है। यह विभिन्न फ़ाइल स्वरूपों से टेक्स्ट और इमेज सामग्री दोनों को निकालकर मल्टीमॉडल इनपुट का समर्थन करता है, जबकि अर्थ संबंधी सुसंगतता बनाए रखने के लिए संदर्भ-जागरूक चंकिंग को नियोजित करता है। निर्माण प्रक्रिया टेम्पलेट-आधारित प्रॉम्प्ट इंजेक्शन द्वारा संचालित होती है, और परिणामी आउटपुट को एक स्वचालित मूल्यांकन सिस्टम के माध्यम से मान्य किया जाता है जो गुणवत्ता और सटीकता सुनिश्चित करने के लिए न्यायाधीशों के रूप में भाषा मॉडल का उपयोग करता है। प्रोजेक्ट डेटा प्रोसेसिंग क्षमताओं की एक विस्तृत श्रृंखला को कवर करता है, जिसमें दस्तावेज़ पार्सिंग, स्वचालित गुणवत्ता फ़िल्टरिंग और स्कीमा-अज्ञेयवादी सीरियलाइज़ेशन शामिल है। यह तर्क ट्रेस और टूल-उपयोग प्रदर्शनों जैसे विविध प्रशिक्षण उदाहरणों के निर्माण का समर्थन करता है, और मशीन लर्निंग प्रशिक्षण फ्रेमवर्क के साथ संगतता के लिए अंतिम डेटासेट को मानकीकृत स्वरूपों में निर्यात करता है। उपयोगकर्ता केंद्रीकृत कॉन्फ़िगरेशन फ़ाइलों और कमांड-लाइन तर्कों के माध्यम से निर्माण वर्कफ़्लो और पाइपलाइन चरणों का प्रबंधन करते हैं।