Awesome GitHub RepositoriesDocument Processing Pipelines

LlamaIndex is a comprehensive development framework designed to connect private or external data sources to large language models. It functions as a data-centric toolkit that enables the construction of retrieval-augmented generation systems, allowing developers to build applications that provide context-aware answers based on specific organizational information. The project distinguishes itself through a robust agentic orchestration engine that supports the creation of autonomous agents capable of multi-step reasoning, memory management, and complex tool execution. Beyond simple retrieval, i

Surya is a document processing platform designed to transform unstructured files into structured, machine-readable data. It provides a comprehensive suite of tools for text recognition, layout analysis, and reading order detection, enabling the conversion of PDFs and images into formats such as JSON, HTML, or markdown. The platform is built to handle complex document workflows, offering capabilities for data extraction, document segmentation, and automated form completion. The platform distinguishes itself through a robust pipeline-based architecture that allows users to chain analysis tasks

JSDoc is a JavaScript API documentation generator that parses comments in source code to produce structured documentation files for a project interface. It functions as a source code documentation tool that extracts metadata from code comments to automate the creation of technical API references. The system operates as a template-based documentation engine, supporting external templates to customize the visual presentation and layout of the output. It also serves as a Markdown documentation exporter, transforming extracted documentation into Markdown files for use on alternative publishing pl

Unstructured is an enterprise-grade data orchestration engine designed to transform raw, unstructured files into structured, machine-readable formats. It functions as a comprehensive platform for document ingestion, partitioning, and enrichment, specifically engineered to prepare complex data for retrieval-augmented generation and agentic AI workflows. The platform distinguishes itself through its sophisticated document processing strategies, which combine rule-based extraction with vision-language models to handle diverse file layouts, tables, and images. It provides a modular architecture t

Vespa is a distributed search engine, vector database, and machine learning ranking engine. It serves as an AI search platform designed to handle large-scale document indexing and complex query processing across a cluster of nodes, combining keyword retrieval with high-dimensional embedding storage for semantic similarity search. The platform distinguishes itself by integrating machine learning models directly into the search pipeline to perform real-time inference and ranking. It converts these models into ranking expressions to score and order results based on relevance, while providing a s

هذا المشروع عبارة عن نظام لالتقاط بيانات التغيير (CDC) وطبقة مزامنة تنقل البيانات من قواعد بيانات MySQL إلى فهارس Elasticsearch. يعمل كأداة تعيين من علائقية إلى مستند، حيث يحول جداول قاعدة البيانات إلى مستندات قابلة للبحث لتمكين تكامل البيانات في الوقت الفعلي والبحث بالنص الكامل. يتميز المزامِن بدعم إلغاء تطبيع البيانات العلائقية، والذي يحول عمليات الربط (joins) من واحد إلى متعدد في قاعدة البيانات إلى هياكل مستندات أب-ابن. كما يسمح بتجميع الجداول المقسمة، باستخدام أنماط التعبير النمطي لتجميع جداول قاعدة بيانات متعددة في فهرس بحث واحد. يغطي النظام تعيين وتحويل البيانات بشكل شامل، بما في ذلك تحويل نوع الحقل، وتعيين المخطط، وتصفية الحقول المتزامنة. يوظف نموذج معالجة قائماً على خط الأنابيب لفك تشفير ودمج الحقول، مستخدماً كلاً من التحميل الأولي القائم على اللقطات (snapshots) للأساسيات وبث سجلات النظام الثنائية للتحديثات في الوقت الفعلي.