Open-source alternatives to ImageBind

BLIP is a vision-language model framework that combines contrastive, matching, and language modeling objectives to align images with text. Built on a multimodal encoder-decoder architecture, it supports distributed data-parallel training with cosine learning rate scheduling and sliding-window metric tracking for training stability. The framework provides capabilities for image captioning, visual question answering, and cross-modal retrieval, scoring semantic alignment between images and text through learned embeddings. It includes toolkits for fine-tuning pre-trained models on custom datasets

InternVL is a vision-language model framework that fuses a visual encoder with a large language model to translate image features into textual tokens for reasoning. It provides a system for multimodal inference and dialogue, enabling the processing of images and text to answer questions or generate descriptions. The project is distinguished by its high-resolution image processing, which uses dynamic tiling to maintain detail for images up to 4K resolution, and its chain-of-thought visual reasoning for solving complex mathematical and spatial problems. It also supports temporal frame sampling

Multimodal is a machine learning library built on PyTorch for training large-scale models that combine text, image, audio, and video data streams. It functions as a deep learning framework dedicated to generative diffusion models, multi-task training, and vision-language tasks. The library supplies modular building blocks, discrete latent codebook quantization, shared-space embeddings, and stackable adapter layers to handle diverse conditional inputs during training and inference. The framework supports specific architectures for diffusion models, text-to-video generation, image-text retrieva

AutoGluon is an automated machine learning framework and multimodal library designed to automate the end-to-end pipeline from data preprocessing to high-accuracy model training and validation. It functions as an automated model trainer for tabular, image, text, and time series data, as well as a tool for time series forecasting and foundation model finetuning. The project is distinguished by its ability to jointly process and fuse different data types, allowing for the construction of multimodal neural networks that integrate images, text, and structured tables. It supports zero-shot inferenc

This project is a transformer-based framework for generating dense and sparse vector embeddings of text and multimodal data. It serves as a library for fine-tuning models to perform semantic similarity tasks, retrieval, and reranking. The system is distinguished by its support for diverse architectural patterns, including bi-encoders for fast similarity search and cross-encoders for high-precision reranking. It provides dedicated pipelines for multimodal embeddings, mapping text and images into a shared vector space, and implements knowledge distillation to compress large models into smaller,

Recognize-anything is a multimodal foundation model designed for image recognition, visual tagging, and the generation of descriptive text captions from visual input. It functions as a multimodal embedding model that maps images and text into a shared vector space to enable cross-modal retrieval and recognition. The system implements zero-shot image classification and open-vocabulary object detection, allowing it to recognize object categories not present in the original training data through custom label embeddings. It also features a visual tagging engine and a captioning system that produc

This project is a research library and toolkit for deep learning computer vision, focused on implementing transformer and mixer-based architectures for image classification. It processes visual data by converting images into sequences of patches, allowing standard attention mechanisms to capture global dependencies without relying on traditional convolutional operations. The framework distinguishes itself through its support for multimodal embedding analysis, which maps images and text into a shared latent vector space. This capability enables zero-shot classification and cross-modal retrieva

zero_nlp is a distributed framework for training and fine-tuning large language models and multimodal architectures. It provides a specialized toolkit for distributed model parallelism, allowing neural network layers and weights to be partitioned across multiple GPU devices to train models that exceed the memory capacity of a single processor. The project distinguishes itself through a combination of high-throughput data pipelines and parameter-efficient tuning. It utilizes multi-threading and memory mapping to preprocess and stream datasets exceeding 100GB and implements memory-saving adapta

LAVIS is a multimodal large language model framework and vision-language model library. It provides tools for training and evaluating models that integrate visual, textual, and audio data, serving as a cross-modal feature extractor and a zero-shot visual reasoning engine. The framework distinguishes itself by using frozen-backbone integration, where pretrained encoders remain non-trainable while lightweight adapter layers are updated. It employs cross-modal feature alignment to map different representations into a shared embedding space and utilizes a modular model wrapper to swap vision and

LLaMA-Adapter is a parameter-efficient fine-tuning framework designed to adapt large language models using a minimal set of trainable parameters. It functions as an instruction tuning tool and a multimodal adapter, allowing pre-trained models to follow human instructions and process non-textual data. The project specializes in the integration of image, video, audio, and sensor data into language models for cross-modal understanding. It enables the customization of LLaMA models through the use of lightweight adapters, which allows for the extraction and storage of learned weights independently

vjepa2 is a joint-embedding predictive architecture and video self-supervised learning framework. It functions as a visual representation learner and a robotic manipulation model designed to learn representations by predicting future latent states without reconstructing pixels. The system enables the pretraining of video encoders that learn temporally consistent features through masked-token prediction and multi-modal tokenization. It further maps these latent embeddings to specific physical movements via action-conditioned post-training to plan and execute robot arm grasping and picking task

moco is a PyTorch implementation of momentum contrast designed for self-supervised visual representation learning. It serves as a research-based framework for extracting high-level image features from unlabeled datasets by maximizing the similarity between different views of the same image. The system utilizes an asymmetric encoder architecture consisting of a fast-learning online encoder and a slow-evolving momentum encoder to stabilize training. It employs a dictionary-based approach that compares query images against a dynamic queue of negative samples to learn distinguishing visual featur

Qwen2.5-VL is an autoregressive multimodal transformer designed to process interleaved sequences of text and visual tokens. It integrates visual feature embeddings into a shared language model space to perform cross-modal reasoning and generate coherent responses or structured layout code. The project distinguishes itself through vision-language-action mapping, allowing it to perceive visual interfaces and translate that perception into actionable commands for operating digital screens and robotic hardware. It employs dynamic-resolution image encoding and temporal-frame video indexing to hand

sam-hq is a collection of pre-trained vision foundation models and adapters designed for high-quality image segmentation, multimodal feature extraction, and depth estimation. It provides a zero-shot vision model capable of performing segmentation and classification across diverse domains without requiring task-specific training. The project features a high-quality image segmentation tool based on the Segment Anything Model that generates precise masks from spatial prompts. It includes a multimodal feature extractor to generate high-dimensional vector embeddings from both image and text inputs

This project is a high-performance BERT embedding service and inference server designed to map text sequences into fixed-length numerical vectors. It functions as a machine learning microservice and distributed model server that decouples request handling from heavy computation. The system utilizes a ZeroMQ messaging infrastructure to provide low-latency communication between distributed clients and the inference server. It incorporates server-side batch processing and GPU workload scaling to maximize hardware utilization and manage high request volumes. The platform supports semantic search

GroundingDINO is a deep learning vision model and open-vocabulary object detector designed to map natural language prompts to spatial coordinates. It functions as a text-to-bounding-box framework that enables zero-shot image localization, allowing the system to identify and locate arbitrary objects without requiring predefined classes or specific training for those categories. The project distinguishes itself by matching visual features to natural language descriptions to achieve open-set visual recognition. It supports text-guided image localization and the isolation of specific objects base

This project is a framework for training and deploying transformer-based models that map text, images, audio, and video into dense or sparse vector representations. It functions as a multimodal embedding library and semantic search engine used to retrieve relevant documents by calculating vector similarity between meanings. The framework provides specialized tools for both cross-encoder reranking, which calculates precise similarity scores to refine search results, and vector quantization to compress embedding vectors for reduced memory usage and increased retrieval speed. The project covers

This project provides a foundational framework and reference implementation for executing causal language modeling and multimodal reasoning on local systems. It includes a set of core components for managing model assets, a fine-tuning framework, and structural definitions required to instantiate transformer-based architectures. The system is distinguished by its ability to process combined text and image inputs through multimodal transformer models for visual reasoning and document analysis. It also supports the deployment of quantized models, reducing memory footprints through low-precision

GLM-4 is a large language model and fine-tuning framework designed for human-like text production, complex reasoning, and multilingual conversation. It functions as a multimodal system capable of processing high-resolution visual content and as a long-context model designed to analyze documents with a context window of up to one million tokens. The project differentiates itself through a function calling interface that enables AI agent development by connecting the model to external APIs and real-time web browsing. It includes specialized capabilities for generating functional programming cod

LASER is a cross-lingual sentence embedding library and multilingual text encoder. It functions as a parallel text mining tool that maps sentences from multiple languages into a shared vector space for similarity and classification tasks. The system converts raw text into fixed-length embeddings, enabling the discovery of translation pairs by calculating the vector distance between sentences. This shared representation allows for cross-lingual document classification, where a model trained on one language can be used to categorize documents in another. The library includes a sentence-piece t

This project is a comprehensive machine learning interview guide and technical study resource designed for individuals preparing for machine learning and AI engineering roles. It provides a collection of materials and practice problems covering core algorithms, theoretical fundamentals, and the implementation of neural network architectures. The resource serves as a technical reference for generative AI development, focusing on the design and optimization of large language models and diffusion systems. It includes frameworks for system design, covering the architecture of production machine l

This repository serves as a comprehensive research platform and toolkit for advancing machine learning, quantum computing, and large-scale scientific data analysis. It provides foundational frameworks for developing complex algorithmic systems, offering the necessary infrastructure for distributed training, computational graph execution, and high-performance model development. The project distinguishes itself by integrating specialized research domains with robust, privacy-preserving methodologies. It supports diverse scientific discovery through tools for quantum simulation, physics-informed

Generating shapes using natural language can enable new ways of imagining and creating the things around us. While significant recent progress has been made in text-to-image generation, text-to-shape generation remains a challenging problem due to the unavailability of paired text and shape data…

Point-e is a system for 3D model synthesis that generates three-dimensional point clouds from natural language descriptions and two-dimensional images. It utilizes diffusion models to synthesize these spatial representations based on text prompts or source images. The project includes specialized tools for refining these outputs, such as a point cloud upsampler to increase the density and resolution of low-resolution models. It also provides a mesh converter that uses distance function regression to transform raw point cloud data into structured 3D meshes. The broader capability surface cove

This repository contains implementation of the models described in the paper arXiv:2106.13043. This work is based on our previous works: ESResNe(X)t-fbsp: Learning Robust Time-Frequency Transformation of Audio (2021). ESResNet: Environmental Sound Classification Based on Visual Domain Models (2020).