Awesome GitHub RepositoriesDocument Processing Pipelines

Tesseract is a neural network-based optical character recognition engine designed to convert scanned images and digital documents into machine-readable, searchable text. It functions as both a command-line utility for automating large-scale digitization workflows and a cross-platform library that can be embedded into desktop, mobile, or server-side applications. By utilizing long short-term memory networks, the engine provides robust text extraction across more than one hundred languages and dozens of scripts. The project distinguishes itself through a sophisticated document layout analysis f

MinerU is a document parsing pipeline designed to transform unstructured files into machine-readable, structured data. It utilizes deep learning models to perform layout analysis, identifying document regions and extracting complex content such as mathematical expressions. By combining these neural network inferences with geometric heuristics, the system reconstructs the reading order and structural hierarchy of documents to ensure accurate data representation. The project distinguishes itself through a multi-stage processing workflow that integrates layout detection, optical character recogn

Docling is a modular framework designed for document parsing, layout analysis, and structured data extraction. It transforms unstructured files and web content into a unified, hierarchical data model that preserves the spatial and semantic relationships between text, tables, images, and layout elements. By normalizing diverse input formats into a consistent internal representation, the library enables uniform processing across various document types. The project distinguishes itself through a schema-driven approach that maps document regions to strongly-typed objects, ensuring data accuracy t

This project is a privacy-first backend service designed to facilitate retrieval-augmented generation by processing local documents into searchable vector representations. It provides a modular architecture that allows users to ingest diverse file formats, manage document metadata, and perform semantic searches to provide context-aware responses for chat and completion requests. The system distinguishes itself through a database-agnostic abstraction layer that supports various storage backends, ranging from local disk storage to enterprise-grade vector databases. It offers flexible deployment

This project is a comprehensive framework for constructing, managing, and evaluating knowledge graphs through multi-agent reasoning and deep search capabilities. It provides an end-to-end pipeline that ingests multi-format documents, extracts entities and relationships based on configurable schemas, and maintains structured knowledge bases to support evidence-based retrieval. The system distinguishes itself through its multi-agent orchestration, which decomposes complex queries into parallel research steps and synthesizes long-form reports. It leverages advanced graph-based techniques, includ

Synthetic data kit este un framework integrat conceput pentru a genera, conserva și formata seturi de date de antrenament pentru modelele de limbaj. Oferă un pipeline end-to-end care transformă documentele sursă brute în date structurate potrivite pentru fine-tuning, raționament și antrenarea modelelor pentru utilizarea instrumentelor. Framework-ul se distinge printr-un motor de orchestrare modular care gestionează întregul ciclu de viață al pregătirii datelor. Suportă input multimodal prin extragerea conținutului text și imagine din diverse formate de fișiere, utilizând în același timp chunking-ul conștient de context pentru a menține coerența semantică. Procesul de generare este condus de injectarea de prompt-uri bazată pe template-uri, iar output-ul rezultat este validat printr-un sistem de evaluare automatizat care utilizează modele de limbaj ca judecători pentru a asigura calitatea și acuratețea. Proiectul acoperă o gamă largă de capabilități de procesare a datelor, inclusiv parsarea documentelor, filtrarea automată a calității și serializarea agnostic-ă de schemă. Suportă crearea diverselor exemple de antrenament, cum ar fi urme de raționament și demonstrații de utilizare a instrumentelor, și exportă seturile de date finale în formate standardizate pentru compatibilitate cu framework-urile de antrenament de machine learning. Utilizatorii gestionează fluxul de lucru de generare și etapele pipeline-ului prin fișiere de configurare centralizate și argumente de linie de comandă.