Awesome GitHub RepositoriesMulti-Modal Embedding Models

ImageBind is a multi-modal embedding model and joint representation learner that maps images, text, audio, and other modalities into a single shared vector space. It functions as a cross-modal retrieval framework designed to bind multiple sensory inputs into one cohesive mathematical embedding. The system uses a contrastive learning architecture to align disparate data types by maximizing the similarity between related samples. This allows the model to perform zero-shot multimodal classification and execute cross-modal data retrieval, such as locating visual content via natural language descr

This repository provides a collection of reference implementations and code examples for training and deploying machine learning models using the MLX framework. It serves as a practical guide for executing distributed training, fine-tuning large language models, converting model weights, and implementing multimodal generative workflows. The project distinguishes itself through specialized examples for local hardware execution, featuring weight quantization to reduce memory usage and low-rank adaptation for parameter-efficient fine-tuning. It also includes scripts for transforming external mod

lmdeploy is a high-performance inference engine and deployment framework for large language models and vision models. It functions as a multi-modal model server and compression toolkit designed to serve models with high throughput and low latency. The system enables the distribution of model services across multiple machines using request-based load balancing and tensor parallelism. It includes specialized tools for model quantization and compression to reduce the memory footprint of weights and caches. The framework covers broad capability areas including production deployment, distributed

GLM-4 is a large language model and fine-tuning framework designed for human-like text production, complex reasoning, and multilingual conversation. It functions as a multimodal system capable of processing high-resolution visual content and as a long-context model designed to analyze documents with a context window of up to one million tokens. The project differentiates itself through a function calling interface that enables AI agent development by connecting the model to external APIs and real-time web browsing. It includes specialized capabilities for generating functional programming cod

Gemma 是一个基于仅解码器（decoder-only）Transformer 架构的开放权重（open-weights）大语言模型系列。这些模型专为文本生成和多模态对话而设计，能够根据文本和视觉输入序列处理并生成响应。 该项目提供了一个可微调的 AI 模型，支持权重调整和低秩自适应（LoRA），以针对特定任务优化性能。它包括对量化权重的支持，以减少内存占用并提高在受限硬件上的推理速度。 其功能涵盖了多模态 AI 集成、通过参数分片进行的内存优化，以及集成外部工具和 API 以检索实时数据。它还支持从文本生成图像以及结构化文本输出的采样。

DeepSeek-VL2 是一个多模态大语言模型和视觉语言系统，旨在分析视觉场景并生成描述性文本。它作为一个视觉问答和视觉定位模型，能够从文档中提取信息，并根据文本描述定位图像中的特定对象或区域。 该项目利用专家混合（mixture-of-experts）架构来处理组合的图像和文本输入。它通过增量预填充（incremental prefilling）针对推理进行了优化，从而降低了硬件上的 GPU 内存需求。 该模型涵盖多模态数据分析和视觉文档理解，包括对图表和布局的解释。它执行视觉推理和定位，以将文本查询与相应的视觉内容进行匹配。

LightGlue 是一个深度学习框架，专为图像对之间的局部特征匹配和高速对应估计而设计。它作为一个计算机视觉匹配模型，用于识别不同视角下的对应关键点。 该系统利用自适应神经网络架构，根据输入的图像对动态修剪自身的深度和宽度，从而优化推理速度。这种方法采用 Transformer 风格的注意力机制和跨图像注意力来计算特征描述符之间的相关性。 匹配过程包括迭代细化循环和动态提前停止机制，一旦达到置信度阈值即停止计算。这些功能支持更广泛的计算机视觉流水线，用于实时图像对齐和神经网络推理优化。

sam-hq 是一系列预训练视觉基础模型和适配器，专为高质量图像分割、多模态特征提取和深度估计而设计。它提供了一个零样本（zero-shot）视觉模型，能够在不同领域执行分割和分类，而无需特定任务的训练。 该项目具有基于 Segment Anything Model 的高质量图像分割工具，可从空间提示生成精确掩码。它包括一个多模态特征提取器，用于从图像和文本输入生成高维向量嵌入，以及一个用于从视觉数据预测距离或冠层高度的卷积工具。 该框架涵盖了广泛的计算机视觉功能，包括图像分类、多分辨率特征提取和图像预处理。它支持通过在自定义数据集上进行微调来实现领域自适应，适用于医学影像和遥感等专业应用。 掩码解码器可以转换为开放格式，以便在具有标准运行时的环境中执行。

VisualGLM-6B 是一个双语多模态大语言模型和视觉语言模型，专为对话任务和视觉理解而设计。它是一个双语 AI 模型，能够处理并生成中文和英文的响应。 该系统是一个量化大语言模型，支持 4 位和 8 位精度，以减少本地部署期间的内存占用和硬件要求。它也是一个参数高效微调模型，允许在不进行完全重训练的情况下调整权重，以使系统适应特定的下游任务。 该项目涵盖了多模态对话 AI 和基于图像的对话，支持分析视觉内容以跨多种语言执行视觉理解任务。其功能包括模型精度量化和针对特定领域的微调，以满足专业应用需求。

Multimodal is a machine learning library built on PyTorch for training large-scale models that combine text, image, audio, and video data streams. It functions as a deep learning framework dedicated to generative diffusion models, multi-task training, and vision-language tasks. The library supplies modular building blocks, discrete latent codebook quantization, shared-space embeddings, and stackable adapter layers to handle diverse conditional inputs during training and inference. The framework supports specific architectures for diffusion models, text-to-video generation, image-text retrieva