Awesome GitHub RepositoriesExtraction Configurations

Docling is a modular framework designed for document parsing, layout analysis, and structured data extraction. It transforms unstructured files and web content into a unified, hierarchical data model that preserves the spatial and semantic relationships between text, tables, images, and layout elements. By normalizing diverse input formats into a consistent internal representation, the library enables uniform processing across various document types. The project distinguishes itself through a schema-driven approach that maps document regions to strongly-typed objects, ensuring data accuracy t

Newspaper is a Python library designed for scraping, parsing, and analyzing web-based information. It functions as a framework for automated news aggregation and large-scale web content extraction, providing tools to download, clean, and structure text, metadata, and media from diverse online sources. The project distinguishes itself through a pipeline-oriented architecture that combines heuristic-based content extraction with natural language processing. It automatically identifies and isolates article bodies from web page boilerplate while simultaneously performing language detection, keywo

Instructor is a framework designed for structured data extraction, validation, and language model integration. It functions as a library that transforms unstructured text into validated, type-safe objects by leveraging schema definitions and model-specific tool-calling capabilities. By acting as a validation middleware, the project ensures that language model outputs strictly conform to defined data structures. The library distinguishes itself through a robust validation-based retry loop that automatically re-submits failed responses with error feedback to iteratively correct schema complianc

Kreuzberg is a document extraction engine that converts PDFs, Office files, images, and over 90 other formats into clean, structured text and metadata. It is built around a compiled Rust core that can be used as a native library, a command-line tool, a REST API server, or a WebAssembly module for browser-based processing. The system is designed to run entirely on self-hosted infrastructure, with no data leaving the user's environment. What distinguishes Kreuzberg is its breadth of integration surfaces and its pipeline architecture. It exposes extraction capabilities through native bindings fo

Proxyman is a cross-platform HTTP debugging proxy that captures, inspects, and modifies HTTP, HTTPS, and WebSocket traffic. It functions as a man-in-the-middle proxy, decrypting SSL/TLS traffic to allow real-time inspection and modification of encrypted requests and responses. The tool is designed for debugging web and mobile applications, with capabilities for API mocking and simulation, scriptable traffic modification, and team collaboration on network logs. What distinguishes Proxyman is its deep integration with mobile and cross-platform development workflows. It provides automated certif

frpc-desktop is a cross-platform desktop application that provides a graphical interface for managing FRP intranet penetration tunnels. It combines FRP binary downloading, visual configuration editing, tunnel lifecycle management, and shareable link generation into a single tool, allowing users to tunnel local network services to the public internet through a relay server without manually editing configuration files or managing binaries. The application wraps a web frontend inside an Electron shell for native window management across operating systems, and handles the full lifecycle of the FR

Data-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets to prepare them for training large language and vision models. It functions as a distributed data pipeline engine that runs processing jobs across Ray clusters, handling billions of samples with automatic operator fusion and adaptive parallelism. The framework provides a library of operators that leverage large language models for semantic extraction, filtering, and data synthesis within processing pipelines. The project distinguishes itself through a YAML-based data recipe sys

PaddleX is a PaddlePaddle-based framework for building, deploying, and fine-tuning AI model pipelines, with pre-built support for computer vision, OCR, document analysis, and time series tasks. It offers a toolkit of ready-to-use pipelines for image classification, object detection, segmentation, and pose estimation, alongside an end-to-end OCR document analysis pipeline that extracts text, tables, formulas, and layout information. The platform also includes a dedicated time series forecasting pipeline for analyzing historical data to detect anomalies, classify patterns, and predict future val

ZenML is an extensible machine learning orchestration framework designed to manage the end-to-end lifecycle of data pipelines and AI agent workflows. It functions as a durable orchestrator that executes machine learning tasks as directed acyclic graphs, ensuring that every step is containerized for consistent performance across local, cloud, and hybrid infrastructure. By decoupling pipeline code from underlying compute and storage backends, the platform allows developers to define infrastructure-agnostic stacks that remain portable across diverse environments. The project distinguishes itself

ZenML is an orchestration platform designed for building, deploying, and monitoring reproducible machine learning pipelines and agentic workflows. It provides a unified framework that manages the entire lifecycle of machine learning assets, from data processing and model training to the deployment of persistent inference services. By decoupling pipeline logic from underlying compute and storage, the platform enables teams to transition workflows seamlessly from local development environments to production-grade cloud infrastructure. The platform distinguishes itself through a service-oriented

sChandra is a document processing platform that converts images, PDFs, Word documents, spreadsheets, and other formats into structured output such as HTML, Markdown, or JSON while preserving layout. It can also extract specific data fields from invoices, contracts, or reports using user-defined JSON schemas, with citations back to source locations. The service supports form filling in PDF and image documents, document generation from Markdown, and extraction of tracked changes from Word files. The platform distinguishes itself with pipeline-based processing chains that combine multiple proces

Expose is a secure tunneling tool that creates public URLs for local development servers, making them accessible from the internet without complex network configuration. It forwards incoming HTTP requests and responses between a public relay server and a local machine, enabling external services to reach a development environment that would otherwise be isolated behind a firewall or VPN. The project distinguishes itself through a comprehensive set of webhook debugging and inspection capabilities. It captures all incoming webhook requests and displays them in a real-time dashboard or command-l

GPAC is an open-source multimedia framework built around a pluggable filter graph pipeline, where modular processing units called filters connect into a directed graph to handle media workflows. At its core, the framework centers all media packaging and manipulation on the ISO Base Media File Format (ISOBMFF), with specialized tools for reading, writing, fragmenting, and encrypting MP4 and related containers. It also provides a declarative scene graph composition system for describing interactive multimedia scenes using MPEG-4 BIFS, X3D, SVG, or VRML syntax, alongside a hardware-accelerated re

طقم البيانات الاصطناعية هو إطار عمل متكامل مصمم لتوليد وتنظيم وتنسيق مجموعات بيانات التدريب لنماذج اللغة. يوفر خط أنابيب متكامل يحول مستندات المصدر الخام إلى بيانات مهيكلة مناسبة للضبط الدقيق، والاستدلال، وتدريب نموذج استخدام الأدوات. يتميز إطار العمل بمحرك تنسيق نمطي يدير دورة حياة إعداد البيانات بالكامل. يدعم المدخلات متعددة الوسائط عن طريق استخراج محتوى النص والصورة من تنسيقات ملفات مختلفة، مع استخدام التقطيع الواعي بالسياق للحفاظ على التماسك الدلالي. يتم دفع عملية التوليد من خلال حقن المطالبة القائم على القالب، ويتم التحقق من المخرجات الناتجة من خلال نظام تقييم مؤتمت يستخدم نماذج اللغة كقضاة لضمان الجودة والدقة. يغطي المشروع مجموعة واسعة من قدرات معالجة البيانات، بما في ذلك تحليل المستندات، وتصفية الجودة المؤتمتة، والتسلسل غير الواعي بالمخطط. يدعم إنشاء أمثلة تدريب متنوعة، مثل آثار الاستدلال وعروض استخدام الأدوات، ويصدر مجموعات البيانات النهائية إلى تنسيقات قياسية للتوافق مع أطر عمل تدريب التعلم الآلي. يدير المستخدمون سير عمل التوليد ومراحل خط الأنابيب من خلال ملفات التكوين المركزية ووسائط سطر الأوامر.