langchain-ai/rag-from-scratch
Rag From Scratch
This project is an educational implementation guide and framework for building Retrieval Augmented Generation systems. It provides a workflow for constructing a knowledge base pipeline that partitions documents, indexes them as vectors, and provides external context for language model prompts.
The system features a document chunking framework that uses recursive character splitting to fit text into model context windows. It includes an in-memory vector store and a similarity search system that retrieves relevant text segments by calculating the mathematical distance between dense embedding vectors.
The project covers the end-to-end RAG pipeline development process, including custom data indexing, vector search implementation, and context management for large language models.
The implementation is provided as a series of Jupyter Notebooks.
Features
- RAG Pipelines - Implements an end-to-end workflow that coordinates document ingestion, embedding, retrieval, and final text generation.
- Retrieval Augmented Generation Guides - Provides a comprehensive, step-by-step educational guide for building a complete Retrieval Augmented Generation pipeline from scratch.
- Context Injection - Provides mechanisms for dynamically inserting retrieved documents into AI model prompts for grounding.
- Retrieval Augmented Generation - Combines large language models with external data retrieval to provide context-aware and accurate answers.
- Text Chunks - Implements strategies for dividing long text into smaller overlapping segments to fit model token limits.
- RAG Data Pipelines - Designs the end-to-end workflow from document partitioning and embedding to context retrieval.
- RAG System Design - Provides a framework for indexing custom data and retrieving context for LLM prompts.
- Dense Embeddings - Transforms raw text into high-dimensional numerical vectors that capture semantic meaning for retrieval.
- Vector Similarity Search - Implements algorithms for calculating the mathematical distance between query and document embeddings.
- Local Knowledge Base Indexers - Converts local documents into a searchable vector database for semantic retrieval by AI models.
- Vector Search - Implements semantic search using numerical embeddings to find the most relevant text segments.
- RAG Implementation Guides - Provides an educational guide for partitioning documents and indexing them as vectors for LLM context.
- LLM Context Preparation - Dynamically prepares and inserts relevant information into LLM prompts to improve response quality.
- Recursive Character Splitting - Uses a recursive hierarchy of delimiters to split text into semantically coherent chunks.
- Vector Stores - Ships a local in-memory data structure for storing and retrieving document embeddings.
- AI and Neural Networks - Implementing retrieval-augmented generation systems.
- Artificial Intelligence - Listed in the “Artificial Intelligence” section of the Build Your Own X awesome list.
