tensorflow/datasets
Datasets
This project is a dataset management framework and cross-framework data loader that provides a unified interface for reading data formats compatible with TensorFlow, JAX, and PyTorch. It serves as a library of curated public datasets provided as data streams and includes tools for building, versioning, and documenting large-scale datasets.
The system differentiates itself through a distributed data processing engine capable of managing massive datasets across clusters using parallelized pipelines. It utilizes builder-based construction to standardize how data is downloaded and prepared, while ensuring transparency through attached metadata for citations, release notes, and licenses.
The project covers broad capability areas including input pipeline optimization via shuffling and batching, schema-driven feature mapping for complex data types, and local-cache file persistence. It also supports dataset splitting and sharding, format conversion, and the export of records to common array and dataframe libraries.
Features
- Cross-Framework Compatibility Layers - Provides a unified interface that wraps framework-specific data formats for compatibility between TensorFlow, PyTorch, and JAX.
- Cross-Framework Data Loaders - Provides a unified interface for reading data formats compatible with TensorFlow, JAX, and PyTorch to simplify model training.
- Custom Dataset Definitions - Supports the creation and versioning of new dataset modules with defined schemas for images, text, and audio.
- Large Scale Dataset Processing - Provides a distributed data processing engine to manage and transform massive datasets across clusters using parallel pipelines.
- Dataset Construction Frameworks - Implements a standardized builder-based system for the construction, preparation, and versioning of diverse datasets.
- Public Datasets - Streams curated public datasets directly into machine learning pipelines with configurable splits and versions.
- Distributed Data Processing Engines - Integrates with parallel processing engines like Apache Beam to process large-scale data across distributed clusters.
- Feature Schemas - Uses predefined feature schemas to map raw data into structured tensors for images, audio, and text.
- Dataset Management Frameworks - Offers a framework for versioning, splitting, and documenting large-scale datasets with standardized metadata and feature schemas.
- ML Dataset Lazy-Loading - Implements deferred loading of records as streams to handle datasets that exceed available system memory.
- Cross-Framework Data Pipelines - Provides a unified interface for reading native data formats across TensorFlow, PyTorch, and JAX for better compatibility.
- Input Pipeline Optimizations - Implements input streams with batching, shuffling, and prefetching to optimize throughput during machine learning model training.
- Curated ML Datasets - Provides a comprehensive collection of curated public datasets as data streams for machine learning pipelines.
- Custom Dataset Builder Modules - Offers specialized builder modules for creating new datasets tailored to images, translation folders, or text formats.
- Parallelized Input Pipelines - Provides parallelized input pipelines that prefetch, transform, and batch data to optimize training throughput.
- Dataset Management Tools - Provides command-line utilities for building and versioning datasets to ensure reproducible data management.
- Training and Testing Splits - Enables segmenting data into training, testing, and validation sets using sharding and split logic.
- Dataset Metadata Schemas - Manages dataset documentation and transparency by attaching citations, release notes, and license metadata.
- File-Based Data Sharding - Partitions large datasets into smaller files to enable parallel loading and distributed processing across compute nodes.
- Local Disk Caching - Caches processed data and remote downloads on the local disk to reduce network latency and compute costs.
- Data Pipeline Optimizations - Optimizes data throughput using prefetching, caching, and parallel execution within the input pipeline.
- Metadata Attachments - Pairs raw data tensors with structured metadata including citations, licenses, and versioning information.
- Deep Learning Frameworks - Standardized data loading for TensorFlow and JAX ecosystems.
