# salesforce/blip

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/salesforce-blip).**

5,676 stars · 757 forks · Jupyter Notebook · bsd-3-clause

## Links

- GitHub: https://github.com/salesforce/BLIP
- awesome-repositories: https://awesome-repositories.com/repository/salesforce-blip.md

## Topics

`image-captioning` `image-text-retrieval` `vision-and-language-pre-training` `vision-language` `vision-language-transformer` `visual-question-answering` `visual-reasoning`

## Description

BLIP is a vision-language model framework that combines contrastive, matching, and language modeling objectives to align images with text. Built on a multimodal encoder-decoder architecture, it supports distributed data-parallel training with cosine learning rate scheduling and sliding-window metric tracking for training stability.

The framework provides capabilities for image captioning, visual question answering, and cross-modal retrieval, scoring semantic alignment between images and text through learned embeddings. It includes toolkits for fine-tuning pre-trained models on custom datasets and training vision-language models from scratch, with support for evaluating caption quality, visual reasoning accuracy, and video-text retrieval performance.

Training workflows incorporate learning rate scheduling with warmup, stepwise decay, and cosine decay, while distributed training metrics are synchronized across GPU workers via all-reduce communication. The system also supports extracting unified multimodal features for downstream tasks and logging training progress with periodic summaries.

## Tags

### Artificial Intelligence & ML

- [Training Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/model-training-frameworks/vision-model-training/vision-language-training/training-frameworks.md) — Provides an open-source framework for training, fine-tuning, and evaluating vision-language models on custom image-text datasets.
- [Encoder-Decoder Architectures](https://awesome-repositories.com/f/artificial-intelligence-ml/vision-transformers/encoder-decoder-architectures.md) — Processes images and text through separate encoders then fuses them in a shared transformer decoder for generation tasks.
- [Multimodal Contrastive Losses](https://awesome-repositories.com/f/artificial-intelligence-ml/contrastive-learning-models/multimodal-contrastive-losses.md) — Combines three training objectives to align image-text pairs, classify matching, and generate fluent captions.
- [Data-Parallel Training](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-training-frameworks/data-parallel-training.md) — Replicates the model across multiple GPUs and synchronizes gradients and metrics using all-reduce communication.
- [Text-to-Image Retrieval](https://awesome-repositories.com/f/artificial-intelligence-ml/image-retrieval-systems/text-to-image-retrieval.md) — Finds images whose content matches a text query using learned embeddings. ([source](https://github.com/salesforce/BLIP#readme))
- [Image-Text Match Ranking](https://awesome-repositories.com/f/artificial-intelligence-ml/image-retrieval-systems/text-to-image-retrieval/image-text-match-ranking.md) — Scores how well an image and a piece of text correspond to each other, returning a compatibility score. ([source](https://github.com/salesforce/BLIP#readme))
- [Image-to-Text Retrieval](https://awesome-repositories.com/f/artificial-intelligence-ml/image-retrieval-systems/text-to-image-retrieval/image-to-text-retrieval.md) — Finds the most relevant captions or descriptions from a collection given an image query. ([source](https://github.com/salesforce/BLIP#readme))
- [Vision-Language Training](https://awesome-repositories.com/f/artificial-intelligence-ml/model-training-frameworks/vision-model-training/vision-language-training.md) — Provides specialized training workflows for models that process both visual and textual data.
- [Cross-Modal Similarity Scoring](https://awesome-repositories.com/f/artificial-intelligence-ml/semantic-analysis-tools/semantic-similarity-calculation/cross-modal-similarity-scoring.md) — Computes cosine similarity between image and text embeddings to rank and retrieve matching pairs from a collection.
- [Visual Question Answering](https://awesome-repositories.com/f/artificial-intelligence-ml/visual-question-answering.md) — Trains vision-language models to answer natural language questions about visual content. ([source](https://github.com/salesforce/BLIP/blob/main/train_vqa.py))
- [Multimodal Feature Extractors](https://awesome-repositories.com/f/artificial-intelligence-ml/feature-extraction/multimodal-feature-extractors.md) — Produces a unified vector representation from an image, text, or both for downstream tasks. ([source](https://github.com/salesforce/BLIP#readme))
- [Stepwise Decay Schedules](https://awesome-repositories.com/f/artificial-intelligence-ml/learning-rate-decay-schedules/stepwise-decay-schedules.md) — Reduces the learning rate by a constant factor each epoch while clamping it above a configured minimum value. ([source](https://github.com/salesforce/BLIP/blob/main/utils.py))
- [Cosine Warmup Schedules](https://awesome-repositories.com/f/artificial-intelligence-ml/learning-rate-warmup-strategies/cosine-warmup-schedules.md) — Ramps the learning rate linearly then decays it along a cosine curve to stabilize early training and converge smoothly.
- [From-Scratch Trainings](https://awesome-repositories.com/f/artificial-intelligence-ml/model-training-frameworks/vision-model-training/vision-language-training/from-scratch-trainings.md) — Trains multimodal models from scratch on large collections of image-caption pairs. ([source](https://github.com/salesforce/BLIP/blob/main/README.md))
- [Vision-Language Fine-Tunings](https://awesome-repositories.com/f/artificial-intelligence-ml/model-training-frameworks/vision-model-training/vision-language-training/vision-language-fine-tunings.md) — Adapts pretrained vision-language models to custom tasks using distributed training on custom data.
- [Caption-Based Training](https://awesome-repositories.com/f/artificial-intelligence-ml/text-model-training/caption-based-training.md) — Trains vision-language models to generate descriptive captions for images using paired image-caption datasets. ([source](https://github.com/salesforce/BLIP/blob/main/train_caption.py))
- [Visual Question Answering Evaluation](https://awesome-repositories.com/f/artificial-intelligence-ml/visual-question-answering-evaluation.md) — Runs trained models on test datasets to generate or rank answers for image-question pairs and collects results for scoring. ([source](https://github.com/salesforce/BLIP/blob/main/train_vqa.py))

### Part of an Awesome List

- [Image Captioning](https://awesome-repositories.com/f/awesome-lists/ai/image-captioning.md) — Generates descriptive natural-language text summarizing the visual content of an input image using a vision-language model. ([source](https://github.com/salesforce/BLIP#readme))
- [Visual Question Answering Libraries](https://awesome-repositories.com/f/awesome-lists/ai/image-captioning/visual-question-answering-libraries.md) — Generates natural-language descriptions and answers visual questions from images using multimodal encoder-decoder architectures.
- [Cross-Modal Retrieval Training](https://awesome-repositories.com/f/awesome-lists/ai/cross-modal-models/cross-modal-retrieval-training.md) — Trains models to align images and text by minimizing contrastive and matching losses for cross-modal retrieval. ([source](https://github.com/salesforce/BLIP/blob/main/train_retrieval.py))
- [Cross-Modal Retrieval Frameworks](https://awesome-repositories.com/f/awesome-lists/ai/cross-modal-retrieval-frameworks.md) — Scores image-text alignment and retrieves matching images or captions using learned multimodal embeddings.
- [Vision Language Models](https://awesome-repositories.com/f/awesome-lists/ai/vision-language-models.md) — Unified encoder-decoder architecture for image-language pre-training.