activeloopai/deeplake
Deeplake
DeepLake is AI data infrastructure consisting of a multimodal data lake, a hybrid search engine, and a serverless vector database. It provides a PostgreSQL-based AI data runtime that combines multimodal storage with streaming pipelines to load and shuffle datasets from cloud storage directly into deep learning training pipelines.
The system utilizes lazy indexing to store and slice images, audio, and video without loading entire files into memory. It enables retrieval-augmented generation by persisting high-dimensional embeddings in a serverless vector store and implementing hybrid search that combines vector similarity with full-text keyword matching.
The project covers a broad capability surface including structured metadata indexing for numeric and JSON fields, cloud-local data synchronization, and visualization tools for inspecting dataset annotations such as bounding boxes and masks.
Features
- Serverless Vector Stores - Provides a serverless vector database for storing high-dimensional embeddings to enable scalable retrieval for language models.
- Multimodal Data Storage - Provides a system for saving and organizing mixed-modality data like audio, text, and images using lazy indexing.
- Data Loaders - Provides built-in loaders to efficiently load and shuffle datasets for deep learning model training.
- AI Data Runtimes - Combines serverless PostgreSQL with multimodal storage to enable scalable retrieval and deep learning training.
