zilliztech/deep-searcher
Deep Searcher
Deep Searcher is an open-source retrieval-augmented generation engine that indexes private documents into a vector database and uses large language models to answer complex questions with cited reasoning. It functions as both a command-line interface and a web API research tool, enabling users to load data and generate comprehensive reports by combining indexed private information with LLM-powered analysis.
The system distinguishes itself through a plugin-based provider architecture that supports multiple embedding models, LLM providers, vector databases, and file loaders as interchangeable components. It offers multi-LLM orchestration, coordinating several large language model services to answer queries by routing requests and aggregating responses, while also providing configurable embedding pipelines and vector database retrieval for similarity search.
The project includes CLI-driven data ingestion for local documents and web content, with support for PDFs and text files, alongside web crawling capabilities. Configuration options allow users to select and authenticate with various embedding, LLM, vector database, file loader, and web crawler providers, while the web API service layer exposes query and data loading functions as HTTP endpoints for programmatic access.
Features
- Private Data Querying - Retrieves relevant information from enterprise internal documents by indexing them into a vector database and querying with natural language.
- Research Engines - An open-source engine that indexes private documents and uses LLMs to answer complex questions with cited reasoning.
- CLI-Based Model Querying - Asks questions in plain language from the command line and receives answers based on indexed private data.
- LLM API Integrations - Integrates multiple large language models such as DeepSeek and OpenAI to answer questions and generate content.
- LLM Orchestration - Coordinates multiple large language model providers to answer queries by routing requests and aggregating responses.
- LLM-Powered Search Interfaces - Provides LLM-powered search over private documents using retrieval-augmented generation.
- Retrieval-Augmented Report Generators - Produces comprehensive written reports by combining retrieved private data with LLM reasoning to answer complex questions.
- LLM Provider Configurations - Selects and authenticates with a large language model service to power question answering and content generation.
- Private Document Search Engines - Enables natural language search and reasoning over private enterprise documents using RAG.
- Document Loading Commands - Loads documents into a vector database by running commands in a terminal.
- Local Document Ingestion - Reads files from local storage, including PDFs and text files, to ingest content into the searchable vector database.
- Vector-Database-Backed Retrievals - Indexes document embeddings in a vector database and retrieves relevant chunks via similarity search for LLM context.
- Vector-Store Augmented Generation - Stores document embeddings in Milvus and queries them with LLMs for grounded, cited answers.
- Document and Web Content Indexers - Ingests documents and crawls web content via CLI commands, then indexes them into the vector database for later querying.
- Vector Databases - Indexes document embeddings in Milvus for similarity search and LLM context retrieval.
- Embedding Model Configurations - Converts text into vector representations using pluggable embedding model services with configurable authentication and model selection.
- Query and Load API Serving - Serves query and data loading functions through a web API for programmatic access.
- Vector Database Configurations - Selects and connects to a supported vector database backend for storing and searching document embeddings.
- Multi-LLM Orchestrators - Coordinates multiple LLMs with indexed private data to produce comprehensive, cited reports.
- Web Content Fetching - Fetches and indexes content from specified URLs using configurable web crawlers for inclusion in the knowledge base.
- Embedding Provider Configurations - Switches the text embedding backend to any supported provider by setting its name and model in the configuration.
- Document Format Loaders - Sets the file loader provider and its options to ingest documents from various formats.
- Plugin-Based Architectures - Loads embedding, LLM, vector database, and file loader implementations as interchangeable plugins through a unified configuration interface.
- RAG Pipeline API Services - Exposes query and data loading functions as HTTP endpoints for programmatic access to the RAG pipeline.
- Research APIs - Exposes deep research capabilities through a CLI and web API for loading data and generating reports.
