rom1504/img2dataset
Img2dataset
img2dataset is a high-performance image dataset pipeline and preprocessing tool designed to download and process millions of images from URLs for machine learning training. It functions as a distributed image downloader and cloud storage data exporter, moving large visual datasets from web sources directly into structured formats.
The system prioritizes high-throughput data acquisition by distributing workloads across multiple CPU cores and machines. It integrates directly with remote cloud storage buckets and employs a manifest-based tracking system to resume interrupted downloads without reprocessing existing data.
The tool provides a full preprocessing suite for machine learning dataset preparation, including image resizing, cropping, and property filtering based on size or aspect ratio. It also verifies image integrity via hash comparison and ensures compliance with robot directives during the scraping workflow.
The project is implemented in Python.
Features
- Image Dataset Scrapers - Automates the collection and organization of categorized image files from web sources for model training.
- Image Data Preprocessing - Includes a preprocessing suite for resizing, cropping, and filtering images to ensure consistent quality for model training.
- Large Scale Collection - Downloads millions of images from web URLs to create structured datasets for machine learning and artificial intelligence.
- Vision Dataset Preparation - Formats and organizes raw image data, including resizing and cropping, for computer vision model training.
- Cloud Storage Clients - Provides a unified interface for managing programmatic read and write operations across local filesystems and remote cloud storage.
- Remote Storage Mounts - Enables direct reading and writing of files to remote cloud storage buckets using standard web address prefixes.
- Distributed Data Processing - Scales image download and processing tasks across multiple compute nodes and CPU cores efficiently.
- Download State Tracking - Tracks processed media identifiers in a state file to prevent redundant network requests and enable download resumption.
- Web Data Scraping - Implements a workflow for fetching large volumes of visual data from the web while respecting crawler directives.
- Batch Image Downloads - Implements high-throughput batch retrieval of millions of images from web URLs.
- Local Multiprocessing - Distributes image downloading and processing tasks across multiple CPU cores to maximize hardware throughput.
- Batch Packaging - Groups processed images and metadata into binary containers to optimize loading speeds for machine learning frameworks.
- Cloud Storage Exporters - Pushes processed image datasets directly to remote object storage buckets for centralized management.
- Dataset Exporters - Exports processed images and metadata into specialized file formats optimized for machine learning tools.
- Distributed Data Workload Scaling - Distributes image downloading and processing tasks across multiple CPU cores and machines to maximize throughput.
- Dimension Resizing - Adjusts image pixel dimensions via cropping or padding to ensure consistency across the dataset.
- In-Memory Streaming - Resizes and filters images in memory before writing to disk to reduce temporary storage overhead.
- Property-Based Filters - Filters images from the dataset based on technical properties such as size and aspect ratio.
- Download Resumption - Provides mechanisms to resume interrupted file transfers and fetch missing images using original inputs.
- Data Loading Extraction - Turn image URLs into datasets.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
