openvla/openvlaFork
Openvla
OpenVLA is a vision-language-action model and framework designed for general-purpose robotic manipulation. It provides a robotic policy training framework and a control inference engine that map visual and textual inputs to robotic control actions, enabling zero-shot instruction following on hardware.
The project includes a robotics dataset pipeline for standardizing diverse trajectory data and managing dataset mixtures. It supports large-scale model training through distributed GPU compute and sharded data parallelism, alongside parameter-efficient adaptation for fine-tuning models to new tasks.
The system handles the full lifecycle of robotic policy development, covering action prediction, tokenized action decoding, and weight conversion for standardized library loading. It also provides capabilities for robot policy deployment via REST APIs to enable control on hardware with limited on-device compute.
Features
- Action Output Models - Maps visual and textual inputs to tokenized action sequences for general-purpose robotic manipulation.
- Vision-Language-Action Training - Implements a complete framework for building vision-language-action models from the ground up using distributed GPU workloads.
- Robotic Trajectory Standardizers - Converts diverse trajectory data into a consistent format of observations, actions, and language instructions.
- Vision-Language Fine-Tunings - Provides a framework for adapting pretrained vision-language-action models to new tasks using parameter-efficient fine-tuning.
- Trajectory Standardization - Provides a pipeline for standardizing diverse robotic trajectory data into a consistent format for training.
- Robotic Control Policies - Deploys trained models as control policies that map sensory inputs to robotic movement.
- Action Command Decoding - Decodes discrete output tokens from the language-style model into continuous robotic control commands.
- Robot Policy Inference - Executes pretrained vision-language-action models to perform zero-shot instruction following on hardware.
- Training Frameworks - Provides an open-source framework for training and deploying vision-language-action models.
- Trajectory Data Pipelines - Ships a pipeline for standardizing diverse trajectory data and managing dataset mixtures for training.
- Vision-Language-Action Controllers - Generates precise robotic control actions by processing combined visual and textual inputs through a trained VLA model.
- Tokenized Action Decoding - Translates tokenized action sequences from the VLA model into continuous commands for physical robot execution.
- Multimodal Action Prediction - Generates precise robotic control commands by processing combined visual and textual inputs.
- Distributed Training - Supports scaling model training across multiple GPUs and compute nodes using sharded data parallelism.
- Robot Policy Training Scaling - Scales the training of large robotic action models across multiple GPUs and compute nodes.
- Fully Sharded Data Parallelism - Uses fully sharded data parallelism to distribute training across multiple GPUs for increased throughput.
- Parameter Efficient Fine-Tuning - Implements parameter-efficient fine-tuning to adapt the model to new tasks with reduced memory.
- Robotic Policy Evaluators - Includes tools for evaluating robotic policies through benchmarks in real-world environments and simulations.
- Dataset Blending - Implements weighted sampling from multiple robotics datasets to control training influence.
- Dataset Configuration Systems - Defines a registry of robotics datasets with sampling weights via configuration files.
- Foundation Models - Generalist vision-language-action model for robotic manipulation tasks.
