awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
tensorflow avatar

tensorflow/datasets

0
View on GitHub↗
www.tensorflow.org/datasets↗

Datasets

This project is a dataset management framework and cross-framework data loader that provides a unified interface for reading data formats compatible with TensorFlow, JAX, and PyTorch. It serves as a library of curated public datasets provided as data streams and includes tools for building, versioning, and documenting large-scale datasets.

The system differentiates itself through a distributed data processing engine capable of managing massive datasets across clusters using parallelized pipelines. It utilizes builder-based construction to standardize how data is downloaded and prepared, while ensuring transparency through attached metadata for citations, release notes, and licenses.

The project covers broad capability areas including input pipeline optimization via shuffling and batching, schema-driven feature mapping for complex data types, and local-cache file persistence. It also supports dataset splitting and sharding, format conversion, and the export of records to common array and dataframe libraries.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Features

  • Cross-Framework Compatibility Layers - Provides a unified interface that wraps framework-specific data formats for compatibility between TensorFlow, PyTorch, and JAX.
  • Cross-Framework Data Loaders - Provides a unified interface for reading data formats compatible with TensorFlow, JAX, and PyTorch to simplify model training.
  • Custom Dataset Definitions - Supports the creation and versioning of new dataset modules with defined schemas for images, text, and audio.
  • Large Scale Dataset Processing - Provides a distributed data processing engine to manage and transform massive datasets across clusters using parallel pipelines.
  • Dataset Construction Frameworks - Implements a standardized builder-based system for the construction, preparation, and versioning of diverse datasets.
  • Public Datasets - Streams curated public datasets directly into machine learning pipelines with configurable splits and versions.
  • Distributed Data Processing Engines - Integrates with parallel processing engines like Apache Beam to process large-scale data across distributed clusters.
  • Feature Schemas - Uses predefined feature schemas to map raw data into structured tensors for images, audio, and text.
  • Dataset Management Frameworks - Offers a framework for versioning, splitting, and documenting large-scale datasets with standardized metadata and feature schemas.
  • ML Dataset Lazy-Loading - Implements deferred loading of records as streams to handle datasets that exceed available system memory.
  • Cross-Framework Data Pipelines - Provides a unified interface for reading native data formats across TensorFlow, PyTorch, and JAX for better compatibility.
  • Input Pipeline Optimizations - Implements input streams with batching, shuffling, and prefetching to optimize throughput during machine learning model training.
  • Curated ML Datasets - Provides a comprehensive collection of curated public datasets as data streams for machine learning pipelines.
  • Custom Dataset Builder Modules - Offers specialized builder modules for creating new datasets tailored to images, translation folders, or text formats.
  • Parallelized Input Pipelines - Provides parallelized input pipelines that prefetch, transform, and batch data to optimize training throughput.
  • Dataset Management Tools - Provides command-line utilities for building and versioning datasets to ensure reproducible data management.
  • Training and Testing Splits - Enables segmenting data into training, testing, and validation sets using sharding and split logic.
  • Dataset Metadata Schemas - Manages dataset documentation and transparency by attaching citations, release notes, and license metadata.
  • File-Based Data Sharding - Partitions large datasets into smaller files to enable parallel loading and distributed processing across compute nodes.
  • Local Disk Caching - Caches processed data and remote downloads on the local disk to reduce network latency and compute costs.
  • Data Pipeline Optimizations - Optimizes data throughput using prefetching, caching, and parallel execution within the input pipeline.
  • Metadata Attachments - Pairs raw data tensors with structured metadata including citations, licenses, and versioning information.
  • Deep Learning Frameworks - Standardized data loading for TensorFlow and JAX ecosystems.
4,575 stars·1,593 forks·Python·Apache-2.0·18 views

Star history

Star history chart for tensorflow/datasetsStar history chart for tensorflow/datasets

How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

Frequently asked questions

What does tensorflow/datasets do?

This project is a dataset management framework and cross-framework data loader that provides a unified interface for reading data formats compatible with TensorFlow, JAX, and PyTorch. It serves as a library of curated public datasets provided as data streams and includes tools for building, versioning, and documenting large-scale datasets.

What are the main features of tensorflow/datasets?

The main features of tensorflow/datasets are: Cross-Framework Compatibility Layers, Cross-Framework Data Loaders, Custom Dataset Definitions, Large Scale Dataset Processing, Dataset Construction Frameworks, Public Datasets, Distributed Data Processing Engines, Feature Schemas.

Which projects share features with tensorflow/datasets?

Projects with overlapping indexed features include: huggingface/hub-docs — This repository serves as the documentation source for the Hugging Face Hub, a collaborative platform designed for… datajuicer/data-juicer — Data-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets… open-mmlab/mmdetection3d — MMDetection3D is an open-source toolbox for 3D perception, providing a unified framework for detecting and segmenting… rerun-io/rerun — Rerun is a multimodal data visualizer and robotics data logger designed for rendering synchronized streams of 3D… conardli/easy-dataset — Easy-dataset is a comprehensive platform designed for the end-to-end management of machine learning datasets,… rucaibox/recbole — RecBole is a PyTorch-based recommendation framework designed for building, training, and evaluating a wide variety of…

Projects sharing features with Datasets

These projects share indexed features with Datasets. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
  • huggingface/hub-docshuggingface avatar

    huggingface/hub-docs

    506View on GitHub↗

    This repository serves as the documentation source for the Hugging Face Hub, a collaborative platform designed for hosting, versioning, and discovering machine learning models, datasets, and interactive applications. It provides the foundational infrastructure for managing machine learning assets through Git-based repositories, which support large file storage, branching, and comprehensive commit history. The platform distinguishes itself by integrating metadata-driven discovery and structured management systems that allow users to attach licensing, task categories, and performance metrics to

    Handlebarshacktoberfestmachine-learning
    View on GitHub↗506
  • datajuicer/data-juicerdatajuicer avatar

    datajuicer/data-juicer

    6,574View on GitHub↗

    Data-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets to prepare them for training large language and vision models. It functions as a distributed data pipeline engine that runs processing jobs across Ray clusters, handling billions of samples with automatic operator fusion and adaptive parallelism. The framework provides a library of operators that leverage large language models for semantic extraction, filtering, and data synthesis within processing pipelines. The project distinguishes itself through a YAML-based data recipe sys

    Pythondatadata-analysisdata-pipeline
    View on GitHub↗6,574
  • open-mmlab/mmdetection3dopen-mmlab avatar

    open-mmlab/mmdetection3d

    6,273View on GitHub↗

    MMDetection3D is an open-source toolbox for 3D perception, providing a unified framework for detecting and segmenting objects in three-dimensional environments. It supports a range of core tasks including monocular 3D object detection from single camera images, LiDAR-based 3D object detection from raw point clouds, and multi-modal fusion that combines camera images with LiDAR data. The toolbox also covers point cloud semantic segmentation, assigning class labels to every point in a scan for scene understanding. The project distinguishes itself through a config-driven pipeline that orchestrate

    Python3d-object-detectionobject-detectionpoint-cloud
    View on GitHub↗6,273
  • rerun-io/rerunrerun-io avatar

    rerun-io/rerun

    10,214View on GitHub↗

    Rerun is a multimodal data visualizer and robotics data logger designed for rendering synchronized streams of 3D spatial data, images, and time-series metrics. It functions as a tool for capturing high-frequency sensor data and AI outputs into a queryable columnar format, providing a dedicated interface for viewing MCAP recording files and analyzing physical environments. The project distinguishes itself as a machine learning dataset streamer, capable of feeding logged recordings directly into GPU buffers and PyTorch training pipelines without intermediate exports. It supports a high-performa

    Rustcomputer-visioncppmultimodal
    View on GitHub↗10,214
Compare all 30 related projects→