Awesome GitHub RepositoriesDocument Processing Pipelines

LlamaIndex is a comprehensive development framework designed to connect private or external data sources to large language models. It functions as a data-centric toolkit that enables the construction of retrieval-augmented generation systems, allowing developers to build applications that provide context-aware answers based on specific organizational information. The project distinguishes itself through a robust agentic orchestration engine that supports the creation of autonomous agents capable of multi-step reasoning, memory management, and complex tool execution. Beyond simple retrieval, i

Surya is a document processing platform designed to transform unstructured files into structured, machine-readable data. It provides a comprehensive suite of tools for text recognition, layout analysis, and reading order detection, enabling the conversion of PDFs and images into formats such as JSON, HTML, or markdown. The platform is built to handle complex document workflows, offering capabilities for data extraction, document segmentation, and automated form completion. The platform distinguishes itself through a robust pipeline-based architecture that allows users to chain analysis tasks

JSDoc is a JavaScript API documentation generator that parses comments in source code to produce structured documentation files for a project interface. It functions as a source code documentation tool that extracts metadata from code comments to automate the creation of technical API references. The system operates as a template-based documentation engine, supporting external templates to customize the visual presentation and layout of the output. It also serves as a Markdown documentation exporter, transforming extracted documentation into Markdown files for use on alternative publishing pl

Unstructured is an enterprise-grade data orchestration engine designed to transform raw, unstructured files into structured, machine-readable formats. It functions as a comprehensive platform for document ingestion, partitioning, and enrichment, specifically engineered to prepare complex data for retrieval-augmented generation and agentic AI workflows. The platform distinguishes itself through its sophisticated document processing strategies, which combine rule-based extraction with vision-language models to handle diverse file layouts, tables, and images. It provides a modular architecture t

Vespa is a distributed search engine, vector database, and machine learning ranking engine. It serves as an AI search platform designed to handle large-scale document indexing and complex query processing across a cluster of nodes, combining keyword retrieval with high-dimensional embedding storage for semantic similarity search. The platform distinguishes itself by integrating machine learning models directly into the search pipeline to perform real-time inference and ranking. It converts these models into ranking expressions to score and order results based on relevance, while providing a s

Dieses Projekt ist ein Change-Data-Capture-System und eine Synchronisationsschicht, die Daten aus MySQL-Datenbanken in Elasticsearch-Indizes verschiebt. Es fungiert als Relational-to-Document-Mapper, der Datenbanktabellen in durchsuchbare Dokumente umwandelt, um Echtzeit-Datenintegration und Volltextsuche zu ermöglichen. Der Synchronizer zeichnet sich durch die Unterstützung der Denormalisierung relationaler Daten aus, die Eins-zu-Viele-Datenbank-Joins in Eltern-Kind-Dokumentstrukturen umwandelt. Er ermöglicht zudem die Aggregation partitionierter Tabellen unter Verwendung regulärer Ausdrücke, um mehrere Datenbanktabellen in einem einzigen Suchindex zu gruppieren. Das System deckt umfassende Datenmapping- und Transformationsfunktionen ab, einschließlich Feldtypkonvertierung, Schema-Mapping und synchronisierter Filterung von Feldern. Es verwendet ein Pipeline-basiertes Verarbeitungsmodell zum Dekodieren und Zusammenführen von Feldern und nutzt sowohl Snapshot-basiertes Initial-Loading für Baselines als auch Binary-Log-Streaming für Echtzeit-Updates.