Awesome GitHub RepositoriesJoint Embedding Spaces

Clip-as-service is a deployable framework for generating multi-modal embeddings and executing neural searches. It provides a vector embedding server and a CLIP embedding API to convert images and text into shared vector representations via network interfaces. The system functions as a multi-modal ranking system and neural search engine, enabling the retrieval of images through text queries or the identification of matching text descriptions for images. It also includes a visual reasoning service used to analyze images and verify object presence, counts, and colors by comparing visual data aga

该项目是一个高性能 BERT 嵌入服务和推理服务器，旨在将文本序列映射为固定长度的数值向量。它作为一个机器学习微服务和分布式模型服务器，将请求处理与繁重的计算解耦。 该系统利用 ZeroMQ 消息基础设施在分布式客户端和推理服务器之间提供低延迟通信。它结合了服务器端批处理和 GPU 工作负载扩展，以最大化硬件利用率并管理高请求量。 该平台通过在共享向量空间内为文本和图像生成跨模态嵌入来支持语义搜索基础设施。这实现了跨模态搜索、内容相关性排名以及基于视觉内容与文本描述之间语义对齐的结果重排序。 该服务可以作为可通过 gRPC、HTTP 或 WebSocket 协议访问的弹性微服务进行部署，具有用于处理大数据集的非阻塞双工流。

ImageBind is a multi-modal embedding model and joint representation learner that maps images, text, audio, and other modalities into a single shared vector space. It functions as a cross-modal retrieval framework designed to bind multiple sensory inputs into one cohesive mathematical embedding. The system uses a contrastive learning architecture to align disparate data types by maximizing the similarity between related samples. This allows the model to perform zero-shot multimodal classification and execute cross-modal data retrieval, such as locating visual content via natural language descr

本项目提供了一个基础框架和参考实现，用于在本地系统上执行因果语言建模和多模态推理。它包含用于管理模型资产的核心组件、微调框架以及实例化 Transformer 架构所需的结构定义。 该系统的特点是能够通过多模态 Transformer 模型处理文本和图像组合输入，以进行视觉推理和文档分析。它还支持部署量化模型，通过低精度技术减少内存占用，从而在边缘设备上实现推理。 该项目涵盖了广泛的能力领域，包括用于领域定制的监督微调和低秩自适应 (LoRA)，以及用于下载、验证和组织模型权重与分词器的综合资产管理器。其他功能还包括多语言文本生成、长上下文处理和视觉语言接地。

vjepa2 is a joint-embedding predictive architecture and video self-supervised learning framework. It functions as a visual representation learner and a robotic manipulation model designed to learn representations by predicting future latent states without reconstructing pixels. The system enables the pretraining of video encoders that learn temporally consistent features through masked-token prediction and multi-modal tokenization. It further maps these latent embeddings to specific physical movements via action-conditioned post-training to plan and execute robot arm grasping and picking task