langchain-ai/rag-from-scratch
Rag From Scratch
This project is an educational implementation guide and framework for building Retrieval Augmented Generation systems. It provides a workflow for constructing a knowledge base pipeline that partitions documents, indexes them as vectors, and provides external context for language model prompts.
The system features a document chunking framework that uses recursive character splitting to fit text into model context windows. It includes an in-memory vector store and a similarity search system that retrieves relevant text segments by calculating the mathematical distance between dense embedding vectors.
The project covers the end-to-end RAG pipeline development process, including custom data indexing, vector search implementation, and context management for large language models.
The implementation is provided as a series of Jupyter Notebooks.
Features
- RAG Pipelines - Implements an end-to-end workflow that coordinates document ingestion, embedding, retrieval, and final text generation.
- Retrieval Augmented Generation Guides - Provides a comprehensive, step-by-step educational guide for building a complete Retrieval Augmented Generation pipeline from scratch.
- Context Injection - Provides mechanisms for dynamically inserting retrieved documents into AI model prompts for grounding.
- Retrieval Augmented Generation - Combines large language models with external data retrieval to provide context-aware and accurate answers.
- Text Chunks - Implements strategies for dividing long text into smaller overlapping segments to fit model token limits.
- RAG Data Pipelines - Designs the end-to-end workflow from document partitioning and embedding to context retrieval.
- RAG System Design - Provides a framework for indexing custom data and retrieving context for LLM prompts.
- Dense Embeddings - Transforms raw text into high-dimensional numerical vectors that capture semantic meaning for retrieval.
- Vector Similarity Search - Implements algorithms for calculating the mathematical distance between query and document embeddings.
- Local Knowledge Base Indexers - Converts local documents into a searchable vector database for semantic retrieval by AI models.
- Vector Search - Implements semantic search using numerical embeddings to find the most relevant text segments.
- RAG Implementation Guides - Provides an educational guide for partitioning documents and indexing them as vectors for LLM context.
- LLM Context Preparation - Dynamically prepares and inserts relevant information into LLM prompts to improve response quality.
- Recursive Character Splitting - Uses a recursive hierarchy of delimiters to split text into semantically coherent chunks.
- Vector Stores - Ships a local in-memory data structure for storing and retrieving document embeddings.
- AI and Neural Networks - Implementing retrieval-augmented generation systems.
- Artificial Intelligence - Listed in the “Artificial Intelligence” section of the Build Your Own X awesome list.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
