facebookresearch/multimodal

Features

• Diffusion Model Frameworks - Provides a collection of modular building blocks and schedules for constructing and training generative diffusion models from scratch.
• Multimodal Model Training - Combines visual, textual, and auditory data streams to train multi-task models at scale using pretrained weights.
• Adapter Layers - Wraps underlying base architectures with modular adapter layers to handle diverse conditional inputs during training and inference.
• Phrase-Specific Isolation - Locates and boxes specific regions in an image corresponding to noun phrases found in text queries.
• Discrete Latent Representations - Encodes high-dimensional data like video into downsampled embedding vectors through nearest-neighbor codebook lookups for generative modeling.
• Codebook Quantization Layers - Encodes high-dimensional inputs into downsampled embedding vectors through nearest-neighbor codebook lookups for generative modeling.
• Denoising Schedulers - Wraps models, predictors, and schedules to handle single-step training updates or multi-step evaluation sampling.
• Diffusion Models - Defines modular architectures like Dalle2 and Latent Diffusion Models along with the necessary building blocks for end-to-end training.
• Text-to-Image Retrieval - Matches text queries to relevant images or images to relevant text captions by embedding both modalities into a shared space.
• Distributed Training - Trains state-of-the-art models across visual, textual, and auditory data streams using configurable distributed training pipelines.
• Multimodal Fine-Tuning - Provides ready-to-run scripts and notebooks for training, fine-tuning, and evaluating multimodal models.
• Modular Neural Network Design - Combines generic layers, loss functions, encoders, and fusion modules to stitch together novel multi-task network designs.
• Multi-Modal Embedding Models - Projects different data modalities into a joint vector space to compute similarity scores for cross-modal retrieval tasks.
• Multimodal Content Retrieval - Embeds video and text into a joint vector space using specialized encoders to compute similarity matches.
• Multimodal Machine Learning - Provides a library for training large-scale multimodal deep learning models that combine text, image, audio, and video data using PyTorch.
• Neural Network Design Frameworks - Supplies modular building blocks like layers, loss functions, and encoders for assembling custom network designs.
• Text-to-Video Generators - Synthesizes video clips autoregressively from text prompts using discrete token representations and transformer decoders.
• Vision-Language Model Architectures - Supplies a set of pretrained architectures and evaluation pipelines for tasks including visual question answering, image-text retrieval, and phrase grounding.
• Visual Question Answering - Processes image inputs alongside natural language questions to generate accurate text answers using combined visual and textual features.
• Denoising Schedulers - Defines noise types and amounts applied at each step of the generative diffusion process using built-in schedule computations.
• Diffusion Pipelines - Applies stackable helper transformations designed to process data dictionaries for diffusion pipelines.
• Phrase Grounding Evaluators - Measures model accuracy in associating image regions with text captions using evaluation protocols.
• Model Evaluation and Tuning - Runs scripts and evaluation pipelines for established architectures to test performance on retrieval, question answering, and generation tasks.
• Multimodal Data Preprocessing - Applies specialized data transformations and preprocessing pipelines tailored to popular vision, text, and audio model architectures.
• Training Config Override Systems - Allows modifying training parameters dynamically using command-line arguments or configuration overrides without altering base code.
• Model Accuracy Evaluators - Measures retrieval accuracy using standard recall metrics against predefined datasets with checkpoint weights.

Star history