salesforce/LAVIS

Features

• Multimodal Frameworks - Provides a comprehensive framework for training and evaluating large language models that integrate visual, textual, and audio data.
• Multimodal Models - Implements neural network architectures that process and align images and text within a shared representation space.
• Adapter Layers - Leverages pretrained, non-trainable vision and language encoders while updating only lightweight adapter layers.
• Cross-Modal Representations - Converts visual and textual data into unified representations for use in classification and similarity tasks.
• Visual-Textual Alignments - Maps visual and textual representations into a shared embedding space to enable semantic similarity and retrieval.
• Feature Extraction Models - Transforms visual and textual data into high-dimensional vector representations for classification and similarity tasks.
• Feature Extractors - Provides tools to extract high-quality vector representations from images and text for downstream multimodal tasks.
• Vision-Language Models - Provides tools to merge visual and textual data to develop models capable of processing both modalities simultaneously.
• Zero-Shot Inference Engines - Predicts labels or answers by calculating the proximity between image embeddings and text-based candidate prompts.
• Cross-Modal Models - Integrates images, video, audio, and 3D data with frozen language models for cross-modality intelligence.
• Unified Interfaces - Provides a consistent interface to extract representations from various vision and language models for downstream tasks.
• Text-to-Image Generators - Creates images from textual descriptions to facilitate zero-shot subject-driven generation and editing.
• Image Description Generation - Produces natural language descriptions for images using pretrained vision-language models.
• Multimodal Dataset Catalogs - Manages the downloading and organization of common multimodal datasets from remote sources or local directories.
• Multimodal Data Preprocessing Utilities - Standardizes the loading and preprocessing of diverse multimodal datasets into consistent formats for training.
• Modular Backbone Architectures - Provides a modular structure to swap different vision and language backbones without altering training logic.
• Projection Layers - Uses linear layers to transform disparate modality dimensions into a unified vector size for joint processing.
• Visual Question Answering - Responds to free-form natural language questions about image content using integrated multimodal models.
• Reasoning Frameworks - Leverages pretrained vision and language models within zero-shot frameworks to answer questions about visual content.
• Dataset Preparation Scripts - Automates the downloading and organization of large collections of language-vision datasets and their annotations.
• Multimodal Dataset Loaders - Includes a dedicated manager for organizing and loading large-scale language-vision datasets and annotations.
• Vision-Language Model Benchmarking - Benchmarks model accuracy and reasoning effectiveness using standardized and customized language-vision datasets.
• Agentic Visual Reasoning - Zero-shot visual question answering with frozen language models.
• Computer Vision - Library for language-vision intelligence tasks.
• Computer Vision Frameworks - Library for language-and-vision intelligence research.
• Foundation Models - Family of open large-scale multimodal foundation models.
• Multimodal Foundation Models - General-purpose library for vision-language instruction tuning.
• Pretrained Language Models - Library for vision-language model research and instruction tuning.
• Vision Language Models - Frameworks for connecting frozen encoders with language models via adapters.

Star history