facebookresearch/mmf

Features

• YAML-Driven Recipe Configurations - Defines model, dataset, and training parameters through composable YAML configuration files as the primary experiment entry point.
• Multimodal Frameworks - Provides a modular framework for building and training vision-and-language models on multimodal datasets.
• Multimodal Model Assemblers - Assembles vision and language encoders with a classifier head to process image and text inputs jointly.
• Dataset Registration Systems - Defines dataset builders, configurations, classes, and metrics for loading and training on new multimodal data.
• Data-Parallel Training - Scales model training across multiple nodes and GPUs using synchronous distributed data parallelism with optimizer state sharding.
• Multimodal Inference Pipelines - Runs a complete vision-and-language model pipeline from data loading through inference using modular components.
• Hyperparameter Configurations - Defines encoder types, dimensions, and classifier settings through structured configuration files.
• Multimodal Dataset Loaders - Downloads, processes, and batches vision-and-language datasets for model training and evaluation.
• Vision-Language Trainers - Supports distributed training, mixed precision, and checkpointing for multimodal vision-and-language models.
• Visual Question Answering Trainers - Trains vision-and-language models on combined QA and visual genome datasets for image question answering.
• Mixed Precision Training - Accelerates training by automatically switching between half-precision and full-precision floating point operations.
• Distributed Mixed-Precision Training - Scales model training across multiple GPUs and nodes with distributed data parallelism and mixed-precision acceleration.
• Distributed Training - Distributes model training across multiple machines using data parallelism for large-scale experiments.
• Mixed-Dataset Training - Combines multiple datasets into a single training run by merging their configurations and data pipelines for multi-task learning.
• Pretrained Checkpoint Inference - Loads pretrained checkpoints and runs inference on datasets for evaluation or submission generation.
• Multimodal Fine-Tuning - Provides configuration-driven fine-tuning of multimodal models by referencing pretrained checkpoints.
• Training and Evaluation Pipelines - Runs a training loop with dataset, optimizer, scheduler, and metrics, then saves the final checkpoint.
• Model Parameter Configurations - Sets model-specific parameters like architecture options and hyperparameters through dedicated configuration files.
• Model Performance Evaluators - Computes standard accuracy and captioning metrics on validation or test splits for multimodal models.
• Vision-Language Training - Trains multimodal models on specified datasets using provided configurations and saves trained weights.
• Multimodal Baseline Training - Provides single-command baseline training for multimodal models with automatic dataset downloads.
• Multimodal Multi-Dataset Trainers - Trains vision-and-language models on multiple datasets simultaneously using built-in reference implementations.
• Pretrained Model Zoos - Bundles reference implementations of state-of-the-art vision-and-language models with pretrained checkpoints for one-command inference.
• Vision-Language Model Zoos - Provides a curated zoo of pretrained vision-and-language models with predefined configurations for selection.
• Training Checkpointers - Restarts an interrupted training run by loading the last saved model state and optimizer state from disk.
• Checkpoint Resumption - Saves model and optimizer state to disk, enabling training to resume from the last checkpoint after interruption.
• Visual Question Answering - Processes visual questions by reading text in images and combining it with visual objects to predict answers.
• Pretrained Checkpoint Fine-Tuning - Loads weights from a model zoo or file and continues training on a new dataset, optionally mapping only specific layers.
• Configuration-Driven Captioning Trainers - Trains vision-language captioning models configured via YAML files and command-line arguments.
• Dataset Configuration Systems - Specifies dataset-specific settings such as annotation paths, image sources, and processor pipelines through a dedicated configuration file.
• Dataset Loading - Loads curated multimodal datasets including question answering, captioning, and visual reasoning benchmarks.
• Modular Data Processor Pipelines - Transforms raw data samples into model-ready tensors through a configurable chain of reusable processors.
• Reusable Transform Pipelines - Transforms raw data into model-ready tensors using configurable processors with a common interface.
• YAML Config Mergers - Builds a final configuration by merging several YAML files in a declared order where later files override earlier ones.
• Pipeline Component Registration - Registers models, datasets, and tasks with a unique key using decorators so they can be referenced by configuration.
• Registry-Based Component Loaders - Discovers models, datasets, and processors at runtime through decorator-based registration for automatic wiring via configuration.
• Command-Line - Overrides any configuration value at runtime by passing individual arguments or a JSON string on the command line.
• YAML Configuration Files - Sets model, task, dataset, and hyperparameters by pointing to structured YAML configuration files.
• Configuration-Driven Pipelines - Assembles training and evaluation pipelines by merging YAML configuration files for model, dataset, and processor settings.
• Loss Metric Optimizer Customizers - Allows replacement of default losses, metrics, optimizers, and schedulers with user-defined implementations.
• Dataset Splitting Utilities - Reserves a configurable percentage of the training set for validation without requiring separate data files.
• Distributed Training Sharding - Distributes optimizer state across multiple GPUs to reduce per-device memory for larger models.
• Decoding Strategy Caption Generators - Ships caption generation with configurable beam search and nucleus sampling decoding strategies.
• Dataset-Scoped Metrics - Implements a registry-based metric class that computes evaluation scores from model outputs and ground truth.
• Decorator-Based Processor Registrations - Makes a new processor class discoverable by the system through a decorator so it can be referenced in configuration.
• Loss Function Customization - Registers a new loss class by subclassing a base module and decorating it with a registry annotation for automatic discovery.
• Named Loss Dictionaries - Returns a dictionary of named loss tensors from a single forward pass for automatic summation.
• Configurable Loss Combinations - Specifies multiple loss functions in configuration files for combined training.
• Multimodal Checkpoint Evaluations - Loads pretrained multimodal checkpoints and runs inference on validation splits to report accuracy.
• Zoo-Key Model Inference Engines - Generates predictions from pretrained multimodal models by specifying zoo keys and dataset configurations.
• Dataset Prediction Generators - Produces output predictions such as CSV submission files by running a loaded checkpoint on a test dataset.
• Competition Prediction Generators - Produces formatted prediction files that can be submitted directly to an external evaluation platform.
• Vision-Language Multi-Task Trainers - Launches distributed training of unified models on configurable sets of vision and language tasks simultaneously.
• Captioning Metric Evaluators - Evaluates trained vision-language models on validation sets and reports standard captioning metrics.
• TextVQA Training Loops - Runs training loops on the TextVQA dataset with automatic data and feature download.
• Validation Evaluators - Evaluates a loaded checkpoint against a validation split to measure performance without updating weights.
• Custom Sample Processors - Transforms raw data samples into model-ready dictionaries using custom callables.
• Research Reproductions - Uses provided configuration files and checkpoint initialization to replicate specific experimental setups from research papers.
• Multi-Dataset Vision-Language Evaluators - Runs a single pretrained model across several vision-and-language datasets and reports accuracy for each task.
• Paper Model Reproductions - Trains vision-and-language models from research papers using the project's training scripts and data loaders.
• Multimodal Competition Submissions - Generates formatted prediction files for external vision-and-language challenge platforms and computes standard accuracy metrics.
• Caption Prediction Exporters - Generates prediction JSON files for captioning models on validation or test sets with a single command.
• Caption Quality Evaluators - Runs evaluation scripts against ground-truth annotations to compute captioning metrics on validation sets.
• Sample Groupers - Collates individual data samples into batches that group tensors and lists by key for model input.
• Processor Pipeline Configurations - Specifies text and answer processors in a configuration file that the system automatically initializes and attaches to the dataset.
• Dataset Downloaders - Downloads precomputed image features and annotation databases for supported datasets into a standard folder structure.
• Composable Image Transforms - Chains multiple image transformations sequentially for preprocessing vision data.
• Paper Implementations - Runs published multimodal models from research papers using built-in implementations and configurations.
• Image Feature Extraction - Runs utility scripts to generate visual features from images using pretrained object detection models.
• Question Answering Accuracy Evaluators - Measures question-answering accuracy by running trained models on validation sets.
• Computer Vision - Modular framework for multimodal vision and language research.
• Natural Language Processing - Multimodal vision and language research framework.

Star history