Awesome GitHub RepositoriesDocument Processing Pipelines

Tesseract is a neural network-based optical character recognition engine designed to convert scanned images and digital documents into machine-readable, searchable text. It functions as both a command-line utility for automating large-scale digitization workflows and a cross-platform library that can be embedded into desktop, mobile, or server-side applications. By utilizing long short-term memory networks, the engine provides robust text extraction across more than one hundred languages and dozens of scripts. The project distinguishes itself through a sophisticated document layout analysis f

MinerU is a document parsing pipeline designed to transform unstructured files into machine-readable, structured data. It utilizes deep learning models to perform layout analysis, identifying document regions and extracting complex content such as mathematical expressions. By combining these neural network inferences with geometric heuristics, the system reconstructs the reading order and structural hierarchy of documents to ensure accurate data representation. The project distinguishes itself through a multi-stage processing workflow that integrates layout detection, optical character recogn

Docling is a modular framework designed for document parsing, layout analysis, and structured data extraction. It transforms unstructured files and web content into a unified, hierarchical data model that preserves the spatial and semantic relationships between text, tables, images, and layout elements. By normalizing diverse input formats into a consistent internal representation, the library enables uniform processing across various document types. The project distinguishes itself through a schema-driven approach that maps document regions to strongly-typed objects, ensuring data accuracy t

This project is a privacy-first backend service designed to facilitate retrieval-augmented generation by processing local documents into searchable vector representations. It provides a modular architecture that allows users to ingest diverse file formats, manage document metadata, and perform semantic searches to provide context-aware responses for chat and completion requests. The system distinguishes itself through a database-agnostic abstraction layer that supports various storage backends, ranging from local disk storage to enterprise-grade vector databases. It offers flexible deployment

This project is a comprehensive framework for constructing, managing, and evaluating knowledge graphs through multi-agent reasoning and deep search capabilities. It provides an end-to-end pipeline that ingests multi-format documents, extracts entities and relationships based on configurable schemas, and maintains structured knowledge bases to support evidence-based retrieval. The system distinguishes itself through its multi-agent orchestration, which decomposes complex queries into parallel research steps and synthesizes long-form reports. It leverages advanced graph-based techniques, includ

Das Synthetic Data Kit ist ein integriertes Framework, das darauf ausgelegt ist, Trainingsdatensätze für Sprachmodelle zu generieren, zu kuratieren und zu formatieren. Es bietet eine End-to-End-Pipeline, die rohe Quelldokumente in strukturierte Daten umwandelt, die für das Fine-Tuning, Reasoning und das Training von Tool-Use-Modellen geeignet sind. Das Framework zeichnet sich durch eine modulare Orchestrierungs-Engine aus, die den gesamten Lebenszyklus der Datenvorbereitung verwaltet. Es unterstützt multimodale Eingaben durch das Extrahieren von Text- und Bildinhalten aus verschiedenen Dateiformaten, während es kontextbewusstes Chunking einsetzt, um die semantische Kohärenz beizubehalten. Der Generierungsprozess wird durch vorlagenbasierte Prompt-Injektion gesteuert, und die resultierende Ausgabe wird durch ein automatisiertes Bewertungssystem validiert, das Sprachmodelle als Richter verwendet, um Qualität und Genauigkeit sicherzustellen. Das Projekt deckt ein breites Spektrum an Datenverarbeitungsfähigkeiten ab, einschließlich Dokumenten-Parsing, automatisierter Qualitätsfilterung und schema-agnostischer Serialisierung. Es unterstützt die Erstellung diverser Trainingsbeispiele, wie Reasoning-Traces und Tool-Use-Demonstrationen, und exportiert die finalen Datensätze in standardisierte Formate für die Kompatibilität mit Machine-Learning-Trainings-Frameworks. Nutzer verwalten den Generierungs-Workflow und die Pipeline-Phasen über zentrale Konfigurationsdateien und Befehlszeilenargumente.