Projects sharing features with Vjepa2

giga-brain-0 is a robot action model framework designed to train and deploy neural networks that map multi-modal sensor data to physical robot control signals. It functions as a robot manipulation controller that processes high-dimensional observations to execute dexterous, long-horizon physical tasks. The project provides a multi-modal robot inference server using a client-server architecture to stream real-time vision and language observations for instant action prediction. It includes an embodiment fine-tuning pipeline to adapt pre-trained base models to specific robot hardware configurati

This project is a self-supervised vision foundation model based on a vision transformer architecture. It is designed to learn dense visual representations from unlabeled images, serving as a general-purpose backbone for a wide variety of downstream vision tasks. The system is distinguished by its use of self-distillation and masked image modeling to extract semantic and geometric features. It also incorporates an image-text alignment model that maps visual embeddings to textual descriptions, enabling zero-shot image recognition, zero-shot segmentation, and cross-modal retrieval. The project

ImageBind is a multi-modal embedding model and joint representation learner that maps images, text, audio, and other modalities into a single shared vector space. It functions as a cross-modal retrieval framework designed to bind multiple sensory inputs into one cohesive mathematical embedding. The system uses a contrastive learning architecture to align disparate data types by maximizing the similarity between related samples. This allows the model to perform zero-shot multimodal classification and execute cross-modal data retrieval, such as locating visual content via natural language descr

This project provides a foundational framework and reference implementation for executing causal language modeling and multimodal reasoning on local systems. It includes a set of core components for managing model assets, a fine-tuning framework, and structural definitions required to instantiate transformer-based architectures. The system is distinguished by its ability to process combined text and image inputs through multimodal transformer models for visual reasoning and document analysis. It also supports the deployment of quantized models, reducing memory footprints through low-precision

Lightly is a self-supervised learning framework and computer vision data curation tool designed to manage large image datasets and train models on unlabeled data. It functions as a PyTorch vision library and dataset management SDK, providing tools to convert raw images into high-dimensional vectors for similarity search, visualization, and feature extraction. The project implements a variety of self-supervised architectures, including MoCo, SimCLR, VICReg, Barlow Twins, and masked image modeling. It distinguishes itself by combining these learning frameworks with active learning capabilities,

This project is a high-performance BERT embedding service and inference server designed to map text sequences into fixed-length numerical vectors. It functions as a machine learning microservice and distributed model server that decouples request handling from heavy computation. The system utilizes a ZeroMQ messaging infrastructure to provide low-latency communication between distributed clients and the inference server. It incorporates server-side batch processing and GPU workload scaling to maximize hardware utilization and manage high request volumes. The platform supports semantic search

Clip-as-service is a deployable framework for generating multi-modal embeddings and executing neural searches. It provides a vector embedding server and a CLIP embedding API to convert images and text into shared vector representations via network interfaces. The system functions as a multi-modal ranking system and neural search engine, enabling the retrieval of images through text queries or the identification of matching text descriptions for images. It also includes a visual reasoning service used to analyze images and verify object presence, counts, and colors by comparing visual data aga

This project is a self-supervised contrastive learning framework designed to train deep learning models to learn visual representations from images without using human-provided labels. It provides a system for developing pretrained visual representation models that can be adapted for downstream computer vision tasks. The framework includes tools for semi-supervised image classification, which combines large unlabeled datasets with small labeled sets to improve accuracy. It also features a linear probe evaluation tool to assess the quality of learned image features by training a simple linear

This project is a PyTorch vision transformer framework designed for self-supervised learning. It implements a model that trains visual representations using a momentum teacher and self-distillation without the need for labeled data. The library functions as an image feature extractor and visual attention visualizer, allowing for the generation of high-dimensional vectors and the rendering of self-attention maps as heatmaps or videos to analyze model focus. It provides comprehensive tools for downstream vision evaluation, including linear probe classification, k-nearest neighbor categorizatio

DINOv2 is a self-supervised vision transformer foundation model designed to generate high-quality visual representations from raw image data. By leveraging large-scale unlabelled datasets, the framework learns to extract robust numerical embeddings that serve as inputs for various machine learning and analysis workflows. The model distinguishes itself through a teacher-student training framework that utilizes centered and sharpened soft probability distributions to align feature maps across multiple image crops. It incorporates a masking strategy that forces the model to reconstruct missing i

vllm-omni is a high-throughput serving engine and distributed inference framework designed for omni-modal models. It serves as a multi-modal model API server capable of generating text, image, video, and audio data, providing a standardized interface for remote client access. The system features a non-autoregressive generation engine for parallel media production and a robot policy inference server that acts as a real-time communication bridge to robotic hardware using specialized protocols. It supports hybrid execution models that combine sequential token generation with parallelized media g

Magic Animate is a diffusion model video generator designed for human image animation. It transforms a static human photo into a temporally consistent video by mapping movements from a reference motion clip, acting as a tool to create realistic animations from a single image. The system ensures visual stability and minimizes flicker through temporal attention injection and motion-controlled noise scheduling. To accelerate the generation of high-resolution video, it includes a distributed GPU inference engine that splits model workloads across multiple graphics cards. The project covers a com

This is a PyTorch library and framework for self-supervised vision learning. It provides an implementation of masked autoencoders and vision transformers designed to learn image representations by reconstructing masked image patches from unlabeled data. The project features a distributed training pipeline that scales workloads across multiple GPU nodes. This infrastructure includes multi-node orchestration and gradient accumulation to manage large batch sizes and coordinate resource requests across clusters. The toolkit covers a complete workflow from self-supervised masked pre-training to d

Moshi is a real-time voice foundation model and speech-to-speech framework designed for bidirectional, low-latency conversations. It functions as a full-duplex voice interface that processes audio and text concurrently in a single stream, enabling natural human-machine dialogue without sequential processing delays. The system utilizes a neural audio codec to compress high-fidelity audio into low-bitrate tokens for efficient transmission. To manage complex responses and reasoning, it employs internal monologue modeling, which generates a hidden stream of thought tokens alongside audible speech

moco is a PyTorch implementation of momentum contrast designed for self-supervised visual representation learning. It serves as a research-based framework for extracting high-level image features from unlabeled datasets by maximizing the similarity between different views of the same image. The system utilizes an asymmetric encoder architecture consisting of a fast-learning online encoder and a slow-evolving momentum encoder to stabilize training. It employs a dictionary-based approach that compares query images against a dynamic queue of negative samples to learn distinguishing visual featur

Corenet is a deep learning training framework and computer vision model library designed for developing neural networks across vision, text, and audio modalities. It functions as a distributed training orchestrator for scaling workloads across multiple compute nodes and provides a multimodal data pipeline for processing image, text, and video data. The project includes a model conversion toolkit for transforming weights and architectures between different machine learning frameworks. It also provides tools for optimizing model performance on Apple Silicon and reducing response latency in gene

mmpretrain is a modular PyTorch computer vision framework designed for developing, training, and benchmarking deep learning architectures. It serves as a comprehensive toolkit for vision tasks, providing a specialized platform for multimodal machine learning and self-supervised learning. The project features a computer vision model zoo containing architectural definitions and pre-trained weights for backbones such as ViT, ConvNeXt, and Swin Transformer. It distinguishes itself through a dedicated self-supervised learning toolkit that implements algorithms like MAE and DINO to train models wit

This project is a PyTorch implementation of a discrete variational autoencoder designed to compress high-resolution imagery into discrete latent representations. It functions as an image autoencoder that encodes visual data into discrete codes and decodes those codes back into reconstructed images. The system utilizes a latent space image compressor to convert images into a compressed, discrete format. This allows for generative image synthesis and the analysis of image compression by transitioning between raw pixels and discrete code sequences. The implementation covers latent space visuali

Swin-Transformer is a deep learning framework designed for training and deploying hierarchical vision transformer models. It serves as a research library and toolkit for computer vision tasks, providing the infrastructure to build models that replace standard convolution operations with sliding window self-attention mechanisms. By utilizing a multi-scale feature hierarchy, the framework enables the processing of visual data at varying resolutions and spatial scales. The project distinguishes itself through its implementation of shifted window partitioning, which facilitates global information

CLIP is a neural network architecture designed to map visual and textual data into a shared latent vector space. By utilizing transformer-based feature extraction and multi-modal tokenization, the system aligns images and natural language strings, enabling cross-modal similarity analysis and semantic classification. The project functions as a zero-shot classification engine, identifying image content by calculating the cosine similarity between visual features and arbitrary text labels without requiring task-specific retraining. Beyond inference, it serves as a research toolkit for evaluating

Taming Transformers is a generative system for high-resolution image synthesis that combines a vector-quantized GAN image encoder with an autoregressive transformer. It utilizes a discrete latent space to represent images as codebook tokens, enabling the production of high-fidelity visuals through a hybrid architecture. The project provides specialized capabilities for layout-based scene synthesis, allowing for the creation of complex images by placing objects according to defined bounding box coordinates. It also includes tools for image inpainting to fill missing sections of an image by ana

This library provides a comprehensive collection of modular building blocks and research-backed architectures for implementing vision transformers within the PyTorch framework. It serves as a centralized repository for constructing, training, and analyzing attention-based models, offering a wide array of specialized variants designed for image classification and visual representation learning. The project distinguishes itself through a focus on architectural efficiency and flexibility, supporting diverse input formats including non-square images and volumetric data like video. It incorporates

This project is a machine learning research automation system designed to manage the full research lifecycle, from idea discovery to final paper submission. It utilizes markdown-based skill templates to execute autonomous research tasks and manage iterative loops of deep review and experimentation. The system distinguishes itself through integrated capabilities for academic communication and integrity auditing. It can automate the generation of LaTeX papers, conference slide decks, and evidence-grounded peer review rebuttals. To ensure rigor, it employs cross-model review routing and adversar

dalle-mini is a text-to-image model and generative AI system designed to transform natural language descriptions into synthetic images. It functions as an image generation training toolkit and a generative model capable of creating visual representations from text prompts. The project provides a containerized deployment for consistent execution across different computing environments. It includes the necessary scripts and configuration files to train custom generative models from datasets. The system utilizes an autoregressive transformer architecture that treats visual data as discrete toke

LLaVA-NeXT is a multimodal large language model framework and training toolkit designed to process interleaved images and video sequences to generate text. It functions as a visual language model that combines vision encoders with language models to perform complex reasoning, question answering, and video understanding. The system is capable of analyzing high-resolution images and temporal video frames to describe events, summarize actions, and reason across multiple visual inputs. It supports the interpretation of documents and charts, spatial environment analysis, and the generation of desc

BertViz is a diagnostic toolset for visualizing attention heads and internal model behaviors to interpret how language models process text. It serves as an interpretability tool and debugger for natural language processing models, specifically providing interactive maps of attention mechanisms within Transformer architectures. The project enables the analysis of token relationships through detailed views of specific attention heads and layers. It supports global attention visualization across all layers, encoder-decoder attention mapping, and the inspection of individual neurons within query

InternVL is a vision-language model framework that fuses a visual encoder with a large language model to translate image features into textual tokens for reasoning. It provides a system for multimodal inference and dialogue, enabling the processing of images and text to answer questions or generate descriptions. The project is distinguished by its high-resolution image processing, which uses dynamic tiling to maintain detail for images up to 4K resolution, and its chain-of-thought visual reasoning for solving complex mathematical and spatial problems. It also supports temporal frame sampling