Awesome GitHub RepositoriesMulti-modal Embedding Generation

Clip-as-service is a deployable framework for generating multi-modal embeddings and executing neural searches. It provides a vector embedding server and a CLIP embedding API to convert images and text into shared vector representations via network interfaces. The system functions as a multi-modal ranking system and neural search engine, enabling the retrieval of images through text queries or the identification of matching text descriptions for images. It also includes a visual reasoning service used to analyze images and verify object presence, counts, and colors by comparing visual data aga

该项目是一个高性能 BERT 嵌入服务和推理服务器，旨在将文本序列映射为固定长度的数值向量。它作为一个机器学习微服务和分布式模型服务器，将请求处理与繁重的计算解耦。 该系统利用 ZeroMQ 消息基础设施在分布式客户端和推理服务器之间提供低延迟通信。它结合了服务器端批处理和 GPU 工作负载扩展，以最大化硬件利用率并管理高请求量。 该平台通过在共享向量空间内为文本和图像生成跨模态嵌入来支持语义搜索基础设施。这实现了跨模态搜索、内容相关性排名以及基于视觉内容与文本描述之间语义对齐的结果重排序。 该服务可以作为可通过 gRPC、HTTP 或 WebSocket 协议访问的弹性微服务进行部署，具有用于处理大数据集的非阻塞双工流。

PyOD is a Python anomaly detection library used to identify outliers in tabular, time series, graph, text, and image data. It provides a collection of algorithms for detecting anomalous data points and includes a unified detector interface that standardizes input and output signatures across its available detection algorithms. The project features a multi-modal outlier detector for identifying anomalies across diverse formats including unstructured text and images, as well as a specialized toolkit for graph-based and time-series anomaly detection. It includes an ensemble framework for combini

ImageBind is a multi-modal embedding model and joint representation learner that maps images, text, audio, and other modalities into a single shared vector space. It functions as a cross-modal retrieval framework designed to bind multiple sensory inputs into one cohesive mathematical embedding. The system uses a contrastive learning architecture to align disparate data types by maximizing the similarity between related samples. This allows the model to perform zero-shot multimodal classification and execute cross-modal data retrieval, such as locating visual content via natural language descr

Muzic 是一个用于 AI 驱动的音乐分析、创作和合成的深度学习平台和框架。它作为一个音乐生成框架和分析工具，利用大型语言模型和自主智能体来编排符号音乐和音频音乐的创作与解读。 该项目以其跨模态能力而著称，将自然语言和符号音乐映射到共享的联合嵌入空间中，用于零样本分类和信息检索。它采用了多种专门的架构，包括用于音频合成的扩散框架、用于长序列结构一致性的双粒度注意力机制，以及结合音乐理论规则与神经网络的混合系统。 该平台涵盖了广泛的功能，包括从文本和歌词生成 MIDI 序列、神经歌声合成以及自动歌词转录。它还提供用于音乐结构建模、基于属性的符号生成以及通过自主智能体编排外部音乐工具的工具。 支持性实用程序包括用于大规模 MIDI 二进制化、数据集编码的数据工程流水线，以及用于旋律音符提取和语音到音素对齐的音频信号处理。

giga-brain-0 is a robot action model framework designed to train and deploy neural networks that map multi-modal sensor data to physical robot control signals. It functions as a robot manipulation controller that processes high-dimensional observations to execute dexterous, long-horizon physical tasks. The project provides a multi-modal robot inference server using a client-server architecture to stream real-time vision and language observations for instant action prediction. It includes an embodiment fine-tuning pipeline to adapt pre-trained base models to specific robot hardware configurati

Stable-Video-Infinity is a video synthesis tool based on Stable Video Diffusion designed for creating long-form animations and consistent visual content. It serves as an AI video extension framework and a conditioned animation synthesizer capable of producing video sequences of arbitrary length. The project enables infinite video extension by bypassing standard model duration constraints through an error-recycling loop. It supports conditioned animation synthesis using external inputs such as image streams, audio files, or skeletal motion data to guide the generation process. The framework i