awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com

Open Source Vision Language Models

Ranking updated Jun 30, 2026

For an open-source vision-language model, the first results are salesforce/lavis (LAVIS is a comprehensive framework and library of pre-trained vision-language models like BLIP and ALBEF that directly supports image captioning, visual question answering, and zero-shot multimodal reasoning, making it a flagship open-source toolbox for your needs), thudm/glm-4 (GLM-4 is an open-weights multimodal language model that processes both text and images, supports fine-tuning and function calling, and is built for reasoning and visual understanding, directly matching the need for an open-source GPT-4 Vision alternative) and vision-cair/minigpt-4. salesforce/blip and vikhyat/moondream round out the shortlist. Compare the match explanations and check the project documentation against your requirements.

Explore open-source models and frameworks designed for advanced image understanding and multimodal visual reasoning tasks.

Open Source Vision Language Models

Find the best repos with AI.We'll search the best matching repositories with AI.
  • salesforce/lavissalesforce avatar

    salesforce/LAVIS

    11,236View on GitHub↗

    LAVIS is a multimodal large language model framework and vision-language model library. It provides tools for training and evaluating models that integrate visual, textual, and audio data, serving as a cross-modal feature extractor and a zero-shot visual reasoning engine. The framework distinguishes itself by using frozen-backbone integration, where pretrained encoders remain non-trainable while lightweight adapter layers are updated. It employs cross-modal feature alignment to map different representations into a shared embedding space and utilizes a modular model wrapper to swap vision and

    LAVIS is a comprehensive framework and library of pre-trained vision-language models (like BLIP and ALBEF) that directly supports image captioning, visual question answering, and zero-shot multimodal reasoning, making it a flagship open-source toolbox for your needs.

    Jupyter NotebookVision-Language ModelsVisual Question AnsweringZero-Shot Inference Engines
    View on GitHub↗11,236
  • thudm/glm-4THUDM avatar

    THUDM/GLM-4

    7,059View on GitHub↗

    GLM-4 is an open weights large language model designed as a multimodal chat system. It functions as a reasoning-focused and multilingual model capable of processing and generating responses across text and visual data types. The model is distinguished by its function-calling capabilities, allowing it to interface with external tools and APIs to execute tasks and retrieve real-time information. It is optimized for complex logical reasoning, mathematical problem solving, and deep research involving long-form content generation. Broad capabilities include multilingual text generation, the creat

    GLM-4 is an open-weights multimodal language model that processes both text and images, supports fine-tuning and function calling, and is built for reasoning and visual understanding, directly matching the need for an open-source GPT-4 Vision alternative.

    PythonModel Fine-TuningOpen-Weights ModelsMultimodal Models
    View on GitHub↗7,059
  • vision-cair/minigpt-4Vision-CAIR avatar

    Vision-CAIR/MiniGPT-4

    25,561View on GitHub↗

    MiniGPT-4 is a multimodal AI framework and large language model that integrates vision encoders with language models to process and reason about combined image and text inputs. It functions as a vision-language model capable of image-based conversational AI, visual question answering, and multimodal logical reasoning. The project utilizes a pretrained vision-language integration strategy that connects a vision encoder to a language model via a linear projection layer. This approach employs frozen-backbone training to align visual representations with linguistic tokens while keeping the primar

    MiniGPT-4 is an open-source multimodal vision-language model that processes text and images together, supporting visual question answering, image captioning, and conversational reasoning, with pre-trained weights and fine-tuning capability — exactly the kind of model this search is after.

    PythonMultimodal Fine-TuningVision-Language Models
    View on GitHub↗25,561
  • salesforce/blipsalesforce avatar

    salesforce/BLIP

    5,676View on GitHub↗

    BLIP is a vision-language model framework that combines contrastive, matching, and language modeling objectives to align images with text. Built on a multimodal encoder-decoder architecture, it supports distributed data-parallel training with cosine learning rate scheduling and sliding-window metric tracking for training stability. The framework provides capabilities for image captioning, visual question answering, and cross-modal retrieval, scoring semantic alignment between images and text through learned embeddings. It includes toolkits for fine-tuning pre-trained models on custom datasets

    BLIP is a multimodal vision-language model framework that handles image captioning, visual question answering, and cross-modal retrieval, with open-source weights, fine-tuning support, and pre-training on large datasets — exactly the kind of image-understanding AI you’re looking for.

    Jupyter NotebookImage CaptioningVisual Question Answering
    View on GitHub↗5,676
  • vikhyat/moondreamvikhyat avatar

    vikhyat/moondream

    9,769View on GitHub↗

    Moondream is a small-scale vision language model designed to reason across images to generate captions and answer natural language questions. It functions as an edge-optimized system capable of performing visual question answering, image captioning, and object detection. The project distinguishes itself through a lightweight architecture designed for local inference on embedded devices, workstations, and air-gapped hardware. It supports the execution of models on local GPUs and Apple Silicon to ensure data privacy and low latency. The system's capabilities include identifying precise object

    Moondream is an open-source vision-language model that generates captions and answers questions about images, supporting local inference and fine-tuning — exactly the multimodal image understanding tool you are looking for.

    PythonImage CaptioningMultimodal Fine-TuningVision-Language Models
    View on GitHub↗9,769
  • clovaai/donutclovaai avatar

    clovaai/donut

    6,926View on GitHub↗

    Donut is an OCR-free document transformer and end-to-end document parser. It functions as a neural network that converts unstructured document images directly into structured data or text without the use of an external optical character recognition engine. The project includes a synthetic document generator to create artificial images and ground-truth labels for training. It employs a transformer model to perform visual question answering and document image classification based on visual layout and text. The system covers several document understanding capabilities, including structured info

    Donut is a multimodal vision-language model specialized in document understanding — it supports visual question answering and captioning on document images, but its domain focus means it may not handle general images like GPT-4 Vision.

    PythonImage-to-Text TransformersModel Fine-TuningMultimodal Fine-Tuning
    View on GitHub↗6,926
  • opengvlab/internvlOpenGVLab avatar

    OpenGVLab/InternVL

    10,169View on GitHub↗

    InternVL is a vision-language model framework that fuses a visual encoder with a large language model to translate image features into textual tokens for reasoning. It provides a system for multimodal inference and dialogue, enabling the processing of images and text to answer questions or generate descriptions. The project is distinguished by its high-resolution image processing, which uses dynamic tiling to maintain detail for images up to 4K resolution, and its chain-of-thought visual reasoning for solving complex mathematical and spatial problems. It also supports temporal frame sampling

    InternVL is an open-source vision-language model that fuses a visual encoder with a large language model for multimodal reasoning, covering image captioning, visual question answering, and high-resolution image processing, directly matching the search for an open-source alternative to GPT-4 Vision.

    PythonImage CaptioningModel Fine-TuningVision-Language Models
    View on GitHub↗10,169
  • haotian-liu/llavahaotian-liu avatar

    haotian-liu/LLaVA

    24,465View on GitHub↗

    LLaVA is a multimodal large language model architecture designed to process and interpret both image and text inputs to generate natural language responses. It functions as a research-oriented platform for visual instruction tuning, providing a framework to align language models with human intent through training on diverse datasets of paired images and text queries. The system distinguishes itself through a specialized vision-language training pipeline that connects visual data to language models using projection layers and instruction-based fine-tuning. It supports distributed inference by

    LLaVA is an open-source multimodal vision-language model that processes both image and text inputs to generate natural language responses, supporting image captioning, visual question answering, fine-tuning, and zero-shot inference — squarely matching the need for a GPT-4 Vision alternative.

    PythonModel Fine-Tuning
    View on GitHub↗24,465
  • mlfoundations/open_clipmlfoundations avatar

    mlfoundations/open_clip

    14,248View on GitHub↗

    Open CLIP is an open source framework for training and deploying Contrastive Language-Image Pre-training models. It serves as a vision-language training framework and multimodal embedding engine that maps images and text into a shared vector space for similarity searches and zero-shot classification. The project provides a toolkit for distributed training of contrastive models and includes an image-to-text generative model for producing natural language descriptions. It supports custom text encoder integration and utilizes teacher-student model distillation to transfer knowledge from large pr

    Open CLIP is a framework for training and deploying contrastive vision-language models like CLIP, a foundational open-source multimodal model that maps images and text into a shared space for zero-shot classification and captioning — it covers image understanding, fine-tuning, and pre-trained weights, though it is more of an embedding engine than a generative VQA system like GPT-4V.

    PythonContrastive Pre-trainingImage CaptioningImage-to-Text Transformers
    View on GitHub↗14,248
  • karpathy/neuraltalk2karpathy avatar

    karpathy/neuraltalk2

    5,588View on GitHub↗

    Neuraltalk2 is a deep learning vision system designed for automatic image captioning. Built with PyTorch, it utilizes a hybrid architecture that combines a convolutional neural network encoder with a recurrent neural network decoder to generate textual descriptions from visual input. The project features a GPU-accelerated training pipeline capable of distributing workloads across multiple graphics processing units through multi-process distribution. It supports the generation of descriptions for both static image files and real-time video streams. The framework includes capabilities for enco

    Neuraltalk2 is an open-source multimodal vision-language model for generating captions from images, squarely fitting the category but limited to captioning rather than broader visual understanding like full question answering or chat.

    Jupyter NotebookImage CaptioningImage Captioning Training
    View on GitHub↗5,588
  • thudm/cogvlm2THUDM avatar

    THUDM/CogVLM2

    2,437View on GitHub↗

    中文版README

    CogVLM2 is an open-source multimodal vision-language model from THUDM that handles image understanding, visual question answering, and image captioning with fine-tuning and zero-shot inference, directly matching the request for a GPT-4 Vision-like model.

    PythonFine-tuned ModelsMultimodal Agents
    View on GitHub↗2,437
  • google-deepmind/gemmagoogle-deepmind avatar

    google-deepmind/gemma

    5,838View on GitHub↗

    Gemma is a family of open-weights large language models based on a decoder-only transformer architecture. These models are designed for text generation and multi-modal conversations, capable of processing and generating responses based on both textual and visual input sequences. The project provides a fine-tunable AI model that supports weight adjustment and low-rank adaptation to specialize performance for particular tasks. It includes support for quantized weights to reduce memory usage and increase inference speed on limited hardware. The capability surface covers multi-modal AI integrati

    Gemma is an open-weights multimodal language model family that processes both text and image inputs, supports fine-tuning and quantized deployment, and is pre-trained for visual tasks—directly matching the need for an open-source vision-language model.

    PythonModel Fine-TuningOpen-Weights Models
    View on GitHub↗5,838
  • qwenlm/qwen-vlQwenLM avatar

    QwenLM/Qwen-VL

    6,725View on GitHub↗

    Qwen-VL is a multimodal large language model that processes both text and images, enabling tasks like image captioning and visual question answering with open-source weights and support for fine-tuning, directly matching the request for an open-source vision-language model.

    PythonVision-Language Models
    View on GitHub↗6,725
  • deepseek-ai/deepseek-vldeepseek-ai avatar

    deepseek-ai/DeepSeek-VL

    4,192View on GitHub↗

    DeepSeek-VL is a multimodal large language model and image-to-text reasoning engine. It functions as a vision-language model and visual question answering system that integrates visual perception with linguistic reasoning to understand and describe images. The project enables multimodal image understanding and document image analysis, specifically processing screenshots of web pages and technical diagrams. It provides capabilities for visual conversational AI, allowing users to interact with visual data to extract insights and perform complex reasoning across different types of visual informa

    DeepSeek-VL is an open-source vision-language model purpose-built for real-world image understanding, covering tasks like captioning and visual question answering, which directly matches your search for a multimodal AI model similar to GPT-4 Vision.

    PythonVision-Language ModelsVision-Language ModelsVisual Question Answering
    View on GitHub↗4,192
  • internlm/internlm-xcomposerInternLM avatar

    InternLM/InternLM-XComposer

    2,924View on GitHub↗

    InternLM-XComposer-2.5

    InternLM-XComposer-2.5 is an open-source vision-language model built to handle multimodal input (text images) and perform tasks like image understanding and reasoning, which directly matches the query for a GPT-4 Vision–like model, though the sparse description leaves some specifics on features like fine-tuning and inference servers unconfirmed.

    PythonVision Language Models
    View on GitHub↗2,924
  • microsoft/llava-medmicrosoft avatar

    microsoft/LLaVA-Med

    2,214View on GitHub↗

    Large Language-and-Vision Assistant for Biomedicine, built towards multimodal GPT-4 level capabilities.

    LLaVA-Med is a multimodal vision-language model fine-tuned for biomedicine, capable of understanding and analyzing medical images alongside text, which aligns with the goal of an image-understanding AI, though its domain specialization makes it a narrower fit than a general-purpose model.

    PythonMedical Multimodal ModelsMedical Vision Language ModelsPre-training Datasets
    View on GitHub↗2,214
Compare the top 10 at a glance
RepositoryStarsLanguageLicenseLast push
salesforce/lavis11.2KJupyter NotebookBSD-3-ClauseJun 2, 2026
thudm/glm-47.1KPythonApache-2.0Jul 4, 2025
vision-cair/minigpt-425.6KPythonBSD-3-ClauseSep 2, 2024
salesforce/blip5.7KJupyter Notebookbsd-3-clauseAug 5, 2024
vikhyat/moondream9.8KPythonApache-2.0Apr 20, 2026
clovaai/donut6.9KPythonMITJul 11, 2024
opengvlab/internvl10.2KPythonMITSep 22, 2025
haotian-liu/llava24.5KPythonapache-2.0Aug 12, 2024
mlfoundations/open_clip14.2KPythonNOASSERTIONNov 6, 2026
karpathy/neuraltalk25.6KJupyter Notebook—Nov 7, 2017

Related searches

  • Vision language models
  • an open source model for image generation
  • a model for universal image segmentation
  • an open source alternative to Midjourney
  • a model for AI image captioning
  • an open source alternative to OpenAI API
  • an open source platform for LLM hosting
  • an open-source Copilot alternative