Awesome GitHub RepositoriesVision-Language Training

This project is a deep learning image restoration tool designed to remove scratches, fading, and noise from aged photographs and film. It utilizes generative adversarial networks for image translation, alongside specialized networks for face enhancement and video colorization. The system distinguishes itself through a combination of latent-space domain mapping and progressive face enhancement to recover blurred or missing high-frequency facial details. For video content, it employs a colorization framework that uses optical flow and temporal guidance to propagate color from selected keyframes

Open CLIP is an open source framework for training and deploying Contrastive Language-Image Pre-training models. It serves as a vision-language training framework and multimodal embedding engine that maps images and text into a shared vector space for similarity searches and zero-shot classification. The project provides a toolkit for distributed training of contrastive models and includes an image-to-text generative model for producing natural language descriptions. It supports custom text encoder integration and utilizes teacher-student model distillation to transfer knowledge from large pr

Depth-Anything is a monocular depth estimation foundation model that produces dense per-pixel depth maps from a single RGB image. It is built on a DINOv2 Vision Transformer encoder backbone and trained on 62 million unlabeled images using a teacher-student pseudo-labeling framework, enabling robust generalization across diverse scenes without task-specific training. The model outputs both relative depth maps, which capture the ordering of scene points, and metric depth maps with real-world units after fine-tuning on datasets like NYUv2 or KITTI. The project distinguishes itself through its ab

LARK is a development toolkit for training, fine-tuning, and deploying large language models and multimodal models based on PaddlePaddle. It functions as a comprehensive framework that includes an LLM training orchestrator, an inference server, and a multimodal model framework for processing text, image, and video inputs. The project features a retrieval-augmented generation system for building conversational applications that integrate web search and private knowledge bases. It provides specific capabilities for multimodal reasoning and complex logic, enabling the extraction of structured da

CogVLM is a multimodal large language model designed to integrate visual and textual data for reasoning about images and generating natural language. It functions as a visual question answering system that analyzes image content to provide detailed descriptions or answer specific questions. The project includes a visual grounding model capable of mapping text descriptions to precise bounding box coordinates within an image. It also features a vision-based automation agent that analyzes screen captures to generate execution plans and interaction coordinates for software interfaces. The system

ByteTrack is a multi-object tracking framework that implements the ByteTrack algorithm, an ECCV 2022 method designed to recover occluded objects and reduce trajectory fragmentation. The core innovation of the project is its association algorithm, which processes every detection box—including low-confidence ones—by using separate high and low score thresholds, Kalman filter motion prediction, and Hungarian algorithm matching to produce consistent object identities across video frames. The project distinguishes itself by its comprehensive approach to handling occlusions and fragmented trajector

YOLO-World is a vision-language framework and open-vocabulary object detection model. It identifies objects in images and video based on free-form text prompts without requiring predefined category labels. The system enables the identification of arbitrary objects by fusing image features with text embeddings. It includes a specialized tool for automated image labeling, which generates bounding box annotations for custom datasets using text-based prompts. The project provides a deployment pipeline for converting models into quantized ONNX and TFLite formats, supporting real-time inference on

VLM-R1 是一个推理视觉语言模型和具身智能框架，旨在将视觉输入和语言指令映射为物理导航路径点和机器人动作。它作为一个多模态策略优化器和开放词汇检测器，能够根据任意自然语言描述定位物体。 该系统的独特之处在于使用思维链推理和强化学习来解决复杂的视觉和空间任务。它利用视频语义记忆系统，该系统采用视觉缓存来维护实时视频历史记录，以实现低延迟交互和持续的时间推理。 该框架涵盖了广泛的功能，包括用于机器人导航的单目路径点映射、用于物体识别的区域标记定位，以及用于多模态推理稳定性的基于策略的监督微调。它还支持开放词汇检测、指代表达理解，以及通过视觉提示检索提取细粒度物体特征。 该项目使用 Python 实现，并支持在 Ascend 硬件上进行推理。

Chinese-CLIP 是一个多模态框架和视觉-语言模型，专为使用中文文本和图像进行跨模态检索和表征生成而设计。它采用对比学习架构，将视觉和文本数据映射到共享向量空间中进行相似度计算。 该系统支持双向搜索，允许进行文本搜图和以图搜文。它还提供零样本图像分类功能，无需特定任务训练即可识别图像中的对象。 该项目包含用于在专用数据集上微调预训练模型的工具，支持分布式训练和对比学习。它还提供将模型权重导出为优化格式的实用程序，以提高生产环境中的推理速度。

MMF is a modular framework for building, training, and evaluating vision-and-language models. It provides a configuration-driven experiment system where model, dataset, and training parameters are defined through composable YAML files, alongside a curated model zoo of pretrained checkpoints for state-of-the-art multimodal architectures. The framework includes a multimodal dataset loader that downloads, processes, and batches vision-and-language data, and a vision-language model trainer supporting distributed training, mixed precision, and checkpoint-based resumption. The framework distinguish

BLIP is a vision-language model framework that combines contrastive, matching, and language modeling objectives to align images with text. Built on a multimodal encoder-decoder architecture, it supports distributed data-parallel training with cosine learning rate scheduling and sliding-window metric tracking for training stability. The framework provides capabilities for image captioning, visual question answering, and cross-modal retrieval, scoring semantic alignment between images and text through learned embeddings. It includes toolkits for fine-tuning pre-trained models on custom datasets

Neuraltalk2 是一个深度学习视觉系统，专为自动图像标注而设计。它基于 PyTorch 构建，利用卷积神经网络编码器与循环神经网络解码器相结合的混合架构，从视觉输入生成文本描述。 该项目具有 GPU 加速的训练流水线，能够通过多进程分发将工作负载分配到多个图形处理器上。它支持为静态图像文件和实时视频流生成描述。 该框架包括编码器微调、带有温度控制的束搜索（beam search）文本采样，以及使用行业标准语言指标来评估标注准确性和流畅性的功能。它还提供了用于数据集预处理、模型检查点持久化以及将预测结果导出为结构化 JSON 文件的工具。 该实现以 Jupyter Notebook 的形式提供。

rllm is an asynchronous reinforcement learning framework for training language agents. It provides a unified pipeline that runs the same agent code for both evaluation and training, automatically capturing traces for gradient computation. The framework supports distributed reinforcement learning across multiple GPUs and nodes using pluggable backends, and executes agents in isolated sandboxes—either locally or in the cloud—for safe and scalable rollout collection. It trains agents built with LangGraph, SmolAgents, OpenAI Agents SDK, or custom frameworks without requiring core logic changes. T

OpenVLA is a vision-language-action model and framework designed for general-purpose robotic manipulation. It provides a robotic policy training framework and a control inference engine that map visual and textual inputs to robotic control actions, enabling zero-shot instruction following on hardware. The project includes a robotics dataset pipeline for standardizing diverse trajectory data and managing dataset mixtures. It supports large-scale model training through distributed GPU compute and sharded data parallelism, alongside parameter-efficient adaptation for fine-tuning models to new ta

EasyR1 是一个用于大型语言模型和视觉语言模型的分布式模型训练系统及强化学习框架。它充当多模态训练器，并实现了近端策略优化 (PPO) 流水线，旨在提升模型处理文本和图像的推理与感知能力。 该系统专注于在多个计算节点之间分配强化学习工作负载，以管理高内存需求。它通过无填充训练和微调来优化硬件利用率，使大型模型能够适配现有的图形处理器。 该框架涵盖了强化学习和奖励模型编排，包括从人类反馈中进行强化学习 (RLHF) 的工作流。其技术栈包括分布式数据并行、混合精度训练以及用于交错文本和图像数据的多模态输入流水线。 该项目包含用于基于检查点的状态恢复的实用程序，并集成了外部日志记录工具以跟踪训练进度和性能指标。

nanoVLM 是一个用于小型视觉语言模型（VLM）的训练框架和工具包。它提供了一个基于 PyTorch 的环境，用于训练和微调模型，以建立图像输入与文本描述之间的关联并生成自然语言回答。 该项目包含一个云模型版本控制工具，用于将模型权重保存并加载到中央仓库，以在不同环境中同步资产。它还具有专门的评估套件，用于根据标准任务数据集衡量视觉语言模型的准确性和可靠性。 该框架通过 VRAM 消耗测量来涵盖 GPU 资源规划，并通过基于检查点的状态持久化和基于批次的内存管理来管理训练稳定性。

Open Flamingo 是一个多模态大语言模型训练框架，旨在将预训练的视觉编码器与语言模型集成。它实现了一种视觉-语言架构，利用交叉注意力层来处理交错的图像和文本序列。 该系统的特点是其少样本（few-shot）多模态学习能力，允许模型使用提示中提供的少量图像-文本示例来适应新的视觉任务。它支持上下文学习和多模态文本生成，用于视觉问答和字幕生成等任务。 该框架包括一个分布式模型训练器，采用数据并行和梯度检查点技术进行跨多个 GPU 的内存优化。它还提供用于分片多模态数据集加载、并行模型评估的实用程序，以及用于托管大规模模型进行推理的基础设施。

R1-V 是一套用于开发多模态模型的工具集，提供了一个低成本的训练环境，旨在优化大型视觉语言模型的推理和反馈循环。它集成了训练框架、微调流水线和性能评估工具。 该项目具有一个强化学习框架，通过基于视觉验证奖励正确输出来提高视觉推理和泛化能力。它还包括一个监督微调流水线，用于使用标记数据集和配置文件将视觉语言模型定制到特定任务。 该套件包含专门用于评估模型在计数和几何任务中性能的视觉推理评估工具和数据集。