yichuan-w/LEANN
LEANN
LEANN is a framework for local retrieval augmented generation and vector indexing. It functions as a system for building local knowledge bases and source code search engines that combine large language models with retrieved private data to generate context-aware responses.
The project distinguishes itself through a vision-model based document layout extractor for parsing complex PDF figures and diagrams, and a source code search engine that employs structure-aware chunking to preserve function and class boundaries. It also implements the Model Context Protocol to integrate real-time data sources into the retrieval pipeline.
The system provides hybrid information retrieval combining semantic search, exact keyword matching, and boolean metadata filtering. It supports the indexing of diverse data sources, including web browsing history, communication logs, and technical documentation.
Features
- Retrieval Augmented Generation - Combines retrieved local document context with large language models to generate grounded, evidence-based responses.
- Local RAG Implementations - Builds a retrieval augmented generation system that answers questions using private local datasets and large language models.
- RAG Frameworks - Provides a framework for combining large language models with retrieved local data for context-aware responses.
- Hybrid Search Engines - Integrates vector-based semantic retrieval with traditional keyword-based indexing for precise technical information discovery.
- Local Knowledge Bases - Implements a searchable repository for personal data supporting both semantic and keyword-based retrieval.
- Search & Information Retrieval - Provides a retrieval system combining semantic meaning, exact keyword matching, and metadata filtering.
- Semantic Search Engines - Utilizes vector embeddings to retrieve relevant documents based on the conceptual meaning of a query.
- Vector Indexing - Builds high-dimensional vector indices from text data to enable efficient semantic retrieval.
- Structure-Aware Chunking - Processes codebases using structure-aware chunking to preserve function and class boundaries for technical retrieval.
- Source Code Search Engines - Ships a technical retrieval tool that uses structure-aware chunking to index and search programming codebases.
- Syntax-Aware Chunking - Segments source code into logical units based on function and class boundaries to preserve technical context.
- Model Context Protocol - Implements the Model Context Protocol to connect the retrieval pipeline to real-time external data sources.
- Multimodal Vision Models - Employs multimodal vision models to interpret complex PDF layouts and diagrams for structural text extraction.
- Technical Documentation Retrieval - Indexes source code and complex PDF layouts to provide context-aware retrieval of technical specifications.
- Personal Knowledge Management - Indexes browser history, chat logs, and emails to create a searchable personal knowledge base.
- Vision-Based Document Parsers - Uses vision-language models to extract structured text and diagrams from complex PDF files.
- Multimodal Document Ingestion - Extracts information from documents using vision models to interpret complex layouts, figures, and diagrams.
- Context Injection Protocols - Uses a standardized interface to connect real-time data sources for seamless context injection into the retrieval pipeline.
- Real-Time Data Integration Platforms - Connects live data sources to the retrieval pipeline to ensure model responses reflect the most current information.
- Search Filtering Systems - Refines search results using metadata operators to restrict outputs by date, file type, or custom attributes.
- Text Search - Provides keyword and pattern-based text matching to locate specific phrases or error messages.
- Filtered Similarity Searches - Applies boolean metadata filters on attributes like date and file type to constrain vector similarity search results.
