Datasets is a library designed for the management, processing, and sharing of large-scale data collections for machine learning workflows. It functions as both a data processing framework and a versioning platform, providing tools to organize, filter, and transform massive datasets while ensuring reproducibility across research and development teams. The library distinguishes itself by enabling the handling of datasets that exceed available system memory. It utilizes memory-mapped file access, disk-based caching, and lazy iterative streaming to maintain performance when working with large-sca
Hub is a multimodal AI data lake and vector database designed for storing and querying embeddings, text, audio, and images. It functions as a dataset version control system and a machine learning data streaming engine to support large-scale model training. The system utilizes a serverless PostgreSQL vector store to index high-dimensional embeddings for semantic search. It provides a visual interface for inspecting multimodal datasets and viewing annotations such as bounding boxes and masks. The platform handles cloud-agnostic storage synchronization and implements lazy, compressed data strea
AWS Lambda Container Image with Python Rasterio for querying Cloud Optimised GeoTiffs.
DeepLake is AI data infrastructure consisting of a multimodal data lake, a hybrid search engine, and a serverless vector database. It provides a PostgreSQL-based AI data runtime that combines multimodal storage with streaming pipelines to load and shuffle datasets from cloud storage directly into deep learning training pipelines. The system utilizes lazy indexing to store and slice images, audio, and video without loading entire files into memory. It enables retrieval-augmented generation by persisting high-dimensional embeddings in a serverless vector store and implementing hybrid search tha
Intake is a lightweight package for finding, investigating, loading and disseminating data.
Las características principales de intake/intake son: Data Loading Extraction, Gestión de datos.
Las alternativas de código abierto para intake/intake incluyen: huggingface/datasets — Datasets is a library designed for the management, processing, and sharing of large-scale data collections for machine… activeloopai/hub — Hub is a multimodal AI data lake and vector database designed for storing and querying embeddings, text, audio, and… addresscloud/aws-lambda-docker-rasterio — AWS Lambda Container Image with Python Rasterio for querying Cloud Optimised GeoTiffs. ahmed-ali/jsonexport — JSONExport is a multi-language code generator and JSON schema converter that transforms JSON data structures into… ahupp/python-magic — python-magic is a C-binding wrapper that provides a Python interface for the libmagic system library. It functions as… activeloopai/deeplake — DeepLake is AI data infrastructure consisting of a multimodal data lake, a hybrid search engine, and a serverless…