How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
ImageBind is a multi-modal embedding model and joint representation learner that maps images, text, audio, and other modalities into a single shared vector space. It functions as a cross-modal retrieval framework designed to bind multiple sensory inputs into one cohesive mathematical embedding. The system uses a contrastive learning architecture to align disparate data types by maximizing the similarity between related samples. This allows the model to perform zero-shot multimodal classification and execute cross-modal data retrieval, such as locating visual content via natural language descr
Qwen2.5-VL is an autoregressive multimodal transformer designed to process interleaved sequences of text and visual tokens. It integrates visual feature embeddings into a shared language model space to perform cross-modal reasoning and generate coherent responses or structured layout code. The project distinguishes itself through vision-language-action mapping, allowing it to perceive visual interfaces and translate that perception into actionable commands for operating digital screens and robotic hardware. It employs dynamic-resolution image encoding and temporal-frame video indexing to hand
LLaMA-Adapter is a parameter-efficient fine-tuning framework designed to adapt large language models using a minimal set of trainable parameters. It functions as an instruction tuning tool and a multimodal adapter, allowing pre-trained models to follow human instructions and process non-textual data. The project specializes in the integration of image, video, audio, and sensor data into language models for cross-modal understanding. It enables the customization of LLaMA models through the use of lightweight adapters, which allows for the extraction and storage of learned weights independently
LAVIS is a multimodal large language model framework and vision-language model library. It provides tools for training and evaluating models that integrate visual, textual, and audio data, serving as a cross-modal feature extractor and a zero-shot visual reasoning engine. The framework distinguishes itself by using frozen-backbone integration, where pretrained encoders remain non-trainable while lightweight adapter layers are updated. It employs cross-modal feature alignment to map different representations into a shared embedding space and utilizes a modular model wrapper to swap vision and
:construction: WIP :construction:
The main features of descriptinc/lyrebird-wav2clip are: Cross-Modal Models, Representation Learning.
Projects with overlapping indexed features include: facebookresearch/imagebind — ImageBind is a multi-modal embedding model and joint representation learner that maps images, text, audio, and other… salesforce/lavis — LAVIS is a multimodal large language model framework and vision-language model library. It provides tools for training… qwenlm/qwen2.5-vl — Qwen2.5-VL is an autoregressive multimodal transformer designed to process interleaved sequences of text and visual… opengvlab/llama-adapter — LLaMA-Adapter is a parameter-efficient fine-tuning framework designed to adapt large language models using a minimal… yuanzhoulvpi2017/zero_nlp — zero_nlp is a distributed framework for training and fine-tuning large language models and multimodal architectures.… beckschen/transmix — This repository includes the official project for the paper: TransMix: Attend to Mix for Vision Transformers, CVPR 2022.