11 रिपॉजिटरी
End-to-end processes for parsing documents, generating embeddings, and storing chunks for semantic retrieval.
Distinct from Index Construction: Covers the full pipeline from parsing to storage, whereas index construction focuses only on populating the structure.
Explore 11 awesome GitHub repositories matching data & databases · Knowledge Base Construction. Refine with filters or upvote what's useful.
QAnything is a retrieval-augmented generation application framework and self-hosted AI interface. It functions as a system that combines a vector database knowledge base, a document parsing service, and a hybrid search engine to generate answers based on private user data. The project features a modular pipeline architecture that allows users to independently replace components such as parsers, embedding models, and reranking engines. It supports local-first model deployment and offline operation to ensure data privacy, and includes a two-stage retrieval pipeline that merges dense vector embe
Implements end-to-end processes for parsing documents, generating embeddings, and storing chunks for semantic retrieval.
Redis is a high-performance in-memory key-value store that functions as a distributed cache, message broker, and NoSQL database. It provides sub-millisecond read and write access to data stored in RAM and can operate as a vector database for indexing high-dimensional embeddings. The system supports a wide range of data storage and synchronization primitives, including the management of strings, hashes, lists, sets, and JSON documents. It enables real-time data operations through atomic transactions, hybrid persistence using snapshots and append-only logs, and high-availability configurations
Parses documents and creates vector embeddings to build searchable knowledge bases for semantic search.
Skill Seekers is a toolset for generating large language model knowledge bases, featuring a multi-source content scraper and a dedicated RAG data pipeline. It extracts technical data from documentation, code, and video to create structured assets and configuration files for AI-powered IDE extensions. The project distinguishes itself through the ability to transform raw data into polished tutorials and specialized skills for AI plugin marketplaces. It utilizes abstract syntax tree parsing and optical character recognition to analyze GitHub repositories, PDFs, and video frames, converting these
Converts documentation and diverse data sources into structured formats for retrieval pipelines and vector databases.
This project is a collection of tutorials and guides for building large language model applications using the LangChain framework, written in Chinese. It serves as a learning resource for developing software that integrates language models with memory and chain-based logic. The resource provides specific walkthroughs for implementing retrieval augmented generation systems using vector stores and document loaders. It includes guides on creating autonomous agents that dynamically select and execute external tools, as well as tutorials for translating plain text queries into executable database
Walks through the full pipeline of parsing documents and generating embeddings for semantic retrieval.
53AIHub is a centralized orchestration platform for deploying and managing AI agents and prompts across multiple large language model providers. It functions as a multi-model AI gateway and an operation portal for AI services, providing a unified interface to coordinate agents and prompts from various external platforms. The project distinguishes itself as a white-label AI portal designed for self-hosted infrastructure, allowing for full control over operational data on private servers or containers. It includes a comprehensive AI SaaS administration layer with a multi-tenant subscription eng
Implements end-to-end processes for parsing documents and generating embeddings for semantic retrieval.
Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning on Intel CPUs, GPUs, and NPUs. It provides a distributed inference engine for scaling models across multiple accelerators, a multimodal model runtime for vision and speech tasks, and a low-bit model quantization tool for converting weights into INT4, FP8, and GGUF formats. The project features a parameter-efficient finetuning framework that enables model adaptation using QLoRA and DPO on Intel hardware. It distinguishes itself by providing specialized optimizations for Intel XP
Processes knowledge files to create searchable vector-based knowledge bases for question-answering tasks.
DesktopCommanderMCP is a Model Context Protocol (MCP) server that gives AI agents direct access to local files, shell commands, and system processes through natural language instructions. It acts as a unified bridge between conversational commands and desktop operations, enabling an AI to translate plain English into file management, code editing, system command execution, data analysis, and software scaffolding tasks without needing its own API. The server exposes these capabilities as structured tools via the MCP protocol, so any compatible agent can interact with the local environment in a
Creates, updates, and organizes local markdown notes and converts scattered data into structured living documents.
Nexent is an enterprise AI control plane and LLM agent orchestration platform. It provides a zero-code environment for designing, deploying, and managing production AI agents through a multi-agent collaboration framework that coordinates specialized autonomous agents using standardized messaging protocols. The platform integrates the Model Context Protocol to connect agents with external tools, plugins, and services via a universal communication interface. It further distinguishes itself with a dedicated RAG knowledge base manager that imports unstructured documents and utilizes hybrid search
Parses and vectorizes various document formats into searchable knowledge bases with integrated access controls.
Cube Studio एक क्लाउड-नेटिव MLOps प्लेटफॉर्म और Kubernetes-आधारित AI ऑर्केस्ट्रेटर है जिसे पूरे मशीन लर्निंग लाइफसाइकिल के लिए डिज़ाइन किया गया है। यह बड़े पैमाने पर मॉडल फाइन-ट्यूनिंग के लिए एक डिस्ट्रीब्यूटेड ट्रेनिंग फ्रेमवर्क, हार्डवेयर वर्चुअलाइजेशन के लिए GPU रिसोर्स मैनेजर, और एंड-टू-एंड वर्कफ़्लो को मैनेज करने के लिए विजुअल डायरेक्टेड एसाइक्लिक ग्राफ का उपयोग करने वाला ML पाइपलाइन ऑर्केस्ट्रेटर प्रदान करता है। यह प्लेटफॉर्म अपने विशेष LLM इन्फरेंस सर्वर के माध्यम से खुद को अलग बनाता है, जो रिट्रीवल-ऑगमेंटेड जनरेशन और प्राइवेट नॉलेज बेस के निर्माण का समर्थन करता है। इसमें लार्ज लैंग्वेज मॉडल की सुपरवाइज्ड फाइन-ट्यूनिंग और रिइंफोर्समेंट लर्निंग के लिए एक समर्पित सिस्टम है, जो विजुअल हाइपरपैरामीटर सर्च टूल के साथ आता है। सिस्टम मल्टीमॉडल डेटा लेबलिंग, डिस्ट्रीब्यूटेड डेटा पाइपलाइन और मल्टी-क्लस्टर वर्कलोड शेड्यूलिंग जैसी व्यापक परिचालन क्षमताओं को कवर करता है।
Integrates domain-specific data using embeddings and semantic retrieval to build private knowledge bases.
यह प्रोजेक्ट एक .NET Core कंटेंट मैनेजमेंट सिस्टम और मल्टी-साइट प्रबंधन प्लेटफ़ॉर्म है जिसे एक केंद्रीकृत इंटरफ़ेस से स्वतंत्र वेबसाइटों पर संरचित डिजिटल सामग्री को व्यवस्थित और प्रकाशित करने के लिए डिज़ाइन किया गया है। यह एक हेडलेस CMS और स्टेटिक साइट जनरेटर दोनों के रूप में कार्य करता है, जो लोडिंग गति और स्केलेबिलिटी बढ़ाने के लिए डायनामिक टेम्प्लेट को HTML फ़ाइलों में रेंडर करता है। सिस्टम वेबसाइट दस्तावेज़ों और सामग्री को खोजने योग्य AI ज्ञान आधारों में बदलने के लिए रिट्रीवल-ऑगमेंटेड जनरेशन को एकीकृत करता है। इसमें उपयोगकर्ता प्रश्नों और बड़े भाषा मॉडल आउटपुट के बीच लॉजिक को परिभाषित करने के लिए एक विज़ुअल AI वर्कफ़्लो ऑर्केस्ट्रेटर शामिल है। प्लेटफ़ॉर्म स्कीमा-लचीले डेटा मॉडलिंग, डिजिटल एसेट संगठन और भूमिका-आधारित एक्सेस कंट्रोल सहित व्यापक क्षमता क्षेत्रों को कवर करता है। यह विभिन्न उपकरणों के लिए मल्टी-टर्मिनल प्रकाशन, कस्टम मॉड्यूल्स के लिए प्लगइन-आधारित विस्तार आर्किटेक्चर और SM4 डेटा एन्क्रिप्शन और वेब भेद्यता रक्षा की विशेषता वाला एक व्यापक सुरक्षा सूट का समर्थन करता है। सॉफ़्टवेयर को स्व-होस्टेड वेब इंफ्रास्ट्रक्चर के लिए डिज़ाइन किया गया है और इसे Windows, Linux और macOS पर Docker कंटेनरों के माध्यम से परिनियोजित किया जा सकता है।
Implements an end-to-end pipeline for parsing documents and generating embeddings to create searchable AI knowledge bases.
ClaraVerse is a self-hosted orchestration platform for deploying and managing local language models, autonomous agents, and automated workflows on private infrastructure. It functions as a containerized backend manager that orchestrates services, databases, and model providers within local containers to maintain data sovereignty. The platform features a visual workflow builder with a drag-and-drop interface for designing complex parallel task sequences. It utilizes a multi-model abstraction layer to normalize interactions across diverse local and remote AI endpoints and includes a retrieval a
Implements an end-to-end process for parsing documents and generating embeddings for semantic retrieval.