Frameworks de embedding y chunking de documentos
For toolkit para fragmentación (chunking) y embedding de documentos, the strongest matches are openai/chatgpt-retrieval-plugin (This is a full retrieval-augmented generation pipeline that chunks), langchain4j/langchain4j (LangChain4j is a JVM framework for building RAG applications) and bhaskatripathi/pdfgpt (This is a complete RAG pipeline that extracts text). timescale/pgai and zylon-ai/private-gpt round out the shortlist. Each is ranked by relevance to your query, popularity and recent activity.
Herramientas y librerías para dividir documentos de texto en segmentos y generar embeddings vectoriales para búsqueda semántica.
Frameworks de embedding y chunking de documentos
Encuentra los mejores repositorios con IA.Buscaremos los repositorios que mejor coincidan usando IA.
- openai/chatgpt-retrieval-plugin
openai/chatgpt-retrieval-plugin
21,192Ver en GitHubThis project is a retrieval-augmented generation pipeline designed for building custom ChatGPT plugins that allow language models to query private or professional documents. It implements a full retrieval workflow, from processing and indexing document chunks to retrieving relevant context for natural language queries. The system distinguishes itself through a hybrid retrieval approach that combines dense vector embeddings with sparse keyword matching, further refined by a two-stage semantic re-ranking process. It includes specialized data privacy tools for screening personally identifiable i
This is a full retrieval-augmented generation pipeline that chunks documents, generates vector embeddings, and performs hybrid semantic retrieval with re-ranking, exactly matching the need for a document chunking and embedding pipeline for semantic search, and it integrates with LLM frameworks for RAG.
PythonDocument Chunking StrategiesSemantic SearchSemantic Search - langchain4j/langchain4j
langchain4j/langchain4j
12,346Ver en GitHubLangChain4j is a framework and library for building applications powered by large language models on the JVM. It provides a unified API for developing AI agents, implementing retrieval augmented generation, and integrating generative AI capabilities into professional software built with frameworks like Spring Boot or Quarkus. The project enables the creation of autonomous agents that can reason through tasks, manage memory, and execute external tools to achieve specific goals. It differentiates itself through a unified model interface that allows developers to switch between multiple model pr
LangChain4j is a JVM framework for building RAG applications that includes document loaders, text splitters, embedding generation, and vector store integrations (e.g., with Pinecone, Chroma), making it a comprehensive tool for the chunking, embedding, and semantic retrieval pipeline you described.
JavaDocument Parsing PipelinesVector Database AbstractionsVector Database Integrations - bhaskatripathi/pdfgpt
bhaskatripathi/pdfGPT
7,164Ver en GitHubpdfGPT is a retrieval augmented generation application and chatbot designed to analyze PDF documents. It functions as a document analyzer and vector search interface, using large language models to answer questions grounded in the content of uploaded files. The system implements a pipeline that extracts text from PDFs, splits content into overlapping segments, and uses vector-based semantic search to retrieve relevant context. This process allows the application to provide responses with verifiable source citations, including page number references to the original document. The project also
This is a complete RAG pipeline that extracts text from PDFs, splits it into overlapping chunks, generates embeddings, and performs vector similarity search for semantic retrieval — squarely the category the visitor needs, though it currently focuses on PDFs rather than multiple document formats and is packaged as an application rather than a reusable library.
PythonSemantic SearchSemantic SearchText Chunks - timescale/pgai
timescale/pgai
5,802Ver en GitHubpgai is a PostgreSQL AI toolkit and framework designed to integrate large language models and vector embeddings directly into a database. It serves as a bridge for executing machine learning model requests and performing text-to-SQL translations within standard database queries. The project provides an automated vector embedding pipeline that handles the loading, parsing, and chunking of text from tables and unstructured documents. This system utilizes a background worker to synchronize embeddings automatically as source data changes and includes specialized tools for building retrieval-augme
pgai is a PostgreSQL AI toolkit that automatically chunks, embeds, and indexes text from tables and unstructured documents, providing vector similarity search and direct RAG integration — a complete pipeline for semantic retrieval right inside the database.
PLpgSQLSemantic Search EnginesText ChunksVector Embeddings - zylon-ai/private-gpt
zylon-ai/private-gpt
57,278Ver en GitHubThis project is a privacy-first backend service designed to facilitate retrieval-augmented generation by processing local documents into searchable vector representations. It provides a modular architecture that allows users to ingest diverse file formats, manage document metadata, and perform semantic searches to provide context-aware responses for chat and completion requests. The system distinguishes itself through a database-agnostic abstraction layer that supports various storage backends, ranging from local disk storage to enterprise-grade vector databases. It offers flexible deployment
Private-GPT is a privacy-first RAG backend that ingests documents, chunks them, generates embeddings, and enables semantic retrieval over vector stores, matching all required features for document chunking and embedding pipelines.
PythonDocument Parsing PipelinesVector Database AbstractionsVector Database Integrations - truefoundry/cognita
truefoundry/cognita
4,317Ver en GitHubCognita is a retrieval augmented generation orchestration framework used to build pipelines that connect document stores and language models to provide grounded answers. It functions as a document ingestion pipeline and a vector database integrator, managing the process of loading, parsing, and indexing files into a searchable knowledge base. The system includes a language model gateway proxy that provides a unified API to interact with multiple different model providers. This routing layer decouples the application from specific vendors, allowing requests to be proxied through a provider-agn
Cognita is a RAG orchestration framework that provides a document ingestion pipeline for chunking, embedding, and indexing into vector stores, along with a semantic retrieval layer and LLM integration, making it a comprehensive and direct match for building a document chunking and embedding pipeline for semantic search.
PythonVector Database IntegrationsRetrieval-Augmented Generation FrameworksRAG Pipelines - promtengineer/localgpt
PromtEngineer/localGPT
22,215Ver en GitHublocalGPT is a private AI knowledge base and retrieval-augmented generation application. It provides a local document indexer, a hybrid search engine, and an inference interface to enable chatting with private documents and managing a self-hosted information repository without sending data to external servers. The system distinguishes itself through a dual-pass verification pipeline that ensures generated answers are grounded in retrieved sources, accompanied by explicit source attribution. It employs a hybrid retrieval approach combining semantic vector search with keyword matching and rerank
localGPT is a self-contained private AI knowledge base that indexes documents with configurable chunking strategies (including semantic and late chunking), generates embeddings, and performs hybrid semantic-vector plus keyword retrieval, directly matching the need for a document chunking and embedding pipeline with semantic search and RAG integration.
PythonSemantic ChunkingSemantic Search EnginesRetrieval-Augmented Generation - infiniflow/ragflow
infiniflow/ragflow
82,922Ver en GitHubThis project is a comprehensive retrieval-augmented generation platform designed for building, managing, and deploying knowledge-based AI applications. It provides a unified environment for organizing datasets, configuring conversational chat assistants, and developing autonomous agents that execute multi-step reasoning workflows. By integrating document intelligence with advanced retrieval pipelines, the platform enables the creation of grounded, verifiable responses supported by traceable citations. The platform distinguishes itself through deep document understanding and sophisticated know
RAGFlow is a comprehensive RAG platform that natively handles document chunking, embedding generation, and semantic retrieval, making it a direct fit for building a document chunking and embedding pipeline for semantic search—it supports multiple document formats, configurable chunking, and tight integration with LLM frameworks.
PythonDocument Chunking StrategiesDocument Parsing PipelinesSemantic Search Engines - quivrhq/quivr
QuivrHQ/quivr
39,165Ver en GitHubQuivr is a retrieval-augmented generation platform designed to transform raw documents into searchable knowledge bases. It functions as a centralized environment where users can ingest files, index them into vector databases, and interact with language models to receive contextually relevant, data-backed responses. The platform distinguishes itself through an agentic workflow orchestrator that sequences retrieval tasks, tool execution, and model interactions to resolve complex, multi-step queries. This engine is entirely configuration-driven, allowing users to define document ingestion, chunk
Quivr is a retrieval-augmented generation platform that ingests documents, chunks them, indexes them into a vector database, and enables semantic retrieval with LLM integration, directly fitting the need for a document chunking and embedding pipeline.
PythonDocument ParsersRetrieval-Augmented Generation Frameworks - neuml/txtai
neuml/txtai
12,660Ver en GitHubtxtai is an artificial intelligence platform designed for building semantic search applications, managing vector storage, and orchestrating language model workflows. It functions as a comprehensive engine for processing unstructured data, enabling the development of autonomous agents and complex content automation pipelines. The platform distinguishes itself through a hybrid indexing architecture that combines dense vector embeddings with relational graph structures, allowing for multi-dimensional retrieval across both semantic meaning and entity relationships. It supports multimodal analysis
txtai is a comprehensive AI platform that directly provides document chunking, embedding generation, vector storage, and semantic retrieval—exactly the pipeline you need for semantic search over embedded document chunks.
PythonSemantic Search Engines - run-llama/llama_index
run-llama/llama_index
50,306Ver en GitHubLlamaIndex is a comprehensive development framework designed to connect private or external data sources to large language models. It functions as a data-centric toolkit that enables the construction of retrieval-augmented generation systems, allowing developers to build applications that provide context-aware answers based on specific organizational information. The project distinguishes itself through a robust agentic orchestration engine that supports the creation of autonomous agents capable of multi-step reasoning, memory management, and complex tool execution. Beyond simple retrieval, i
LlamaIndex is a comprehensive framework purpose-built for constructing retrieval-augmented generation pipelines—it directly handles document chunking, embedding generation, vector storage integration, and semantic retrieval, covering every feature this query asks for.
PythonRetrieval-Augmented Generation Frameworks - deepset-ai/haystack
deepset-ai/haystack
24,253Ver en GitHubHaystack is an orchestration framework designed for building complex search and generative AI pipelines. It functions as an agentic workflow engine, enabling the construction of automated sequences that allow AI agents to perform multi-step reasoning and data analysis. The framework utilizes a modular, component-based architecture that connects processing steps into directed acyclic graphs. By employing a provider-agnostic integration layer, it decouples core logic from specific external AI services and vector databases, allowing for the flexible exchange of underlying technologies. This desi
Haystack is a modular orchestration framework for building search and RAG pipelines, directly supporting document splitting, embedding generation, vector storage integration, and semantic retrieval across multiple formats, making it a comprehensive solution for your need.
MDXVector Database Integrations - mintplex-labs/anything-llm
Mintplex-Labs/anything-llm
61,663Ver en GitHubThis platform serves as a comprehensive environment for managing private language models, document knowledge bases, and automated agent workflows within secure local infrastructure. It functions as a document-aware workspace that enables users to ingest diverse file formats into searchable repositories, ensuring that all data processing and model inference remain within private, local environments to maintain data sovereignty. The system distinguishes itself through a modular agentic engine that allows for the definition of custom skills and external tool execution. By utilizing a multi-model
mintplex-labs/anything-llm is a complete RAG platform that ingests diverse document formats, splits them into chunks, generates embeddings, and performs vector‑database‑backed semantic retrieval—precisely the document‑aware pipeline you need for building search over embedded chunks.
JavaScriptDocument Parsing Pipelines - supermemoryai/supermemory
supermemoryai/supermemory
27,334Ver en GitHubSupermemory is an artificial intelligence memory management platform designed to provide autonomous agents with persistent, long-term knowledge bases. It functions as a centralized repository that synchronizes multimodal data, enabling agents to maintain context and historical information across complex, multi-session workflows. By serving as a knowledge graph engine and vector database orchestrator, the platform ensures that information remains accessible and relevant for automated tasks. The system distinguishes itself through its hybrid indexing approach, which combines vector similarity s
Supermemory is a platform that provides document chunking, embedding generation, vector storage, and semantic retrieval specifically designed for AI agent memory, covering all the required features including configurable chunking and LLM integration.
TypeScriptSemantic Chunking - hkuds/lightrag
HKUDS/LightRAG
36,651Ver en GitHubLightRAG is a graph-based retrieval framework designed to build retrieval-augmented generation pipelines. It structures unstructured text into knowledge graphs, enabling multi-hop reasoning and complex query synthesis across large document collections. By integrating dense vector embeddings with structured knowledge graphs, the system facilitates both similarity-based and relationship-aware information retrieval. The framework distinguishes itself through a dual-level retrieval strategy that combines low-level keyword matching with high-level semantic graph traversal to capture both specific
LightRAG is a graph-based retrieval framework that ingests documents, structures them into knowledge graphs with dense vector embeddings, and performs dual-level semantic retrieval — this squarely matches the need for a chunking, embedding, and search pipeline, and covers most of the required features for RAG workflows.
PythonKnowledge Graph Retrieval SystemsRetrieval Augmented Generation PipelinesGraph Reasoning Systems - langchain-ai/langchain
langchain-ai/langchain
139,458Ver en GitHubLangChain is an orchestration framework designed for building, managing, and deploying applications powered by large language models. It provides a unified integration layer that normalizes disparate model provider APIs into a consistent set of primitives, enabling developers to build complex, multi-step AI workflows that manage state, memory, and tool execution. The project distinguishes itself through a durable execution runtime that maintains persistent state across long-running processes by checkpointing progress to external storage. It models agent workflows as directed graphs, allowing
LangChain provides all the building blocks—document loaders, text splitters, embedding models, vector store integrations, and retrieval chains—to build exactly the document chunking and semantic search pipeline you're looking for, making it a comprehensive fit.
PythonAgent Orchestration FrameworksLLM Application OrchestrationLLM Integration Layers - zilliztech/claude-context
zilliztech/claude-context
5,373Ver en GitHubClaude-context is a retrieval-augmented generation pipeline and semantic code search tool. It functions as an LLM codebase indexer and RAG context provider, designed to index local directories and retrieve relevant code files to provide context for large language models. The system operates as a hybrid search engine that combines keyword matching with dense vector search. This allows for the retrieval of code snippets and logic using natural language queries based on meaning rather than exact text matches. The project covers codebase indexing and search index management, utilizing asynchrono
Claude-context is a RAG pipeline and semantic search tool specialized for codebases, providing chunking, embedding, and hybrid retrieval — it covers the core chunking-embedding-search workflow and integrates with LLMs, but its focus on code files means it may not handle the broad set of document formats the visitor is looking for.
TypeScriptSemantic Chunking - mayooear/gpt4-pdf-chatbot-langchain
mayooear/gpt4-pdf-chatbot-langchain
16,542Ver en GitHubThis project is a framework for building custom AI chatbots capable of PDF document analysis. It implements Retrieval Augmented Generation to connect a large language model to private document data. The system utilizes graph-based agent orchestration to control conversation flow and decision logic. It maintains context across interactions through thread-based state management and delivers AI responses to the user interface via real-time streaming. The project covers PDF document ingestion through chunk-based processing and vector-store retrieval. It includes mechanisms for query-based data r
This project is a framework for building chatbots over PDFs using chunk-based processing, vector embedding, and RAG retrieval — exactly the document chunking and embedding pipeline you need, though it focuses on PDFs rather than multiple document formats.
TypeScriptDocument Chunking Strategies - microsoft/graphrag
microsoft/graphrag
33,792Ver en GitHubGraphRAG is a data processing pipeline and retrieval engine designed to transform unstructured text into interconnected knowledge graphs. By utilizing language models to extract entities and relationships, it builds structured representations of information that enable context-aware retrieval for downstream applications. The system distinguishes itself through hierarchical graph clustering and large-scale data synthesis, which organize massive document corpora into multi-level structures. This approach allows for both vector-based semantic searches and graph-based traversals, providing a comp
GraphRAG is a pipeline that chunks documents, generates embeddings, and enables semantic retrieval (including vector search) — exactly the kind of tool this search targets, even though its graph-focused approach adds complexity beyond basic chunking.
PythonGraph-Based Retrieval AugmentationGraph-Based Retrieval EnginesContext-Aware Retrieval
