Physical-Intelligence/openpi
Openpi
OpenPi is a vision-language-action robot control framework designed to generate physical control actions for robotic systems. It functions as a distributed robot model trainer, a model format converter, and a robot action streaming server.
The framework provides tools for transforming model checkpoints between different framework formats to ensure interoperability across various development environments. It also includes a server that uses websocket connections to stream model-generated control actions from remote inference servers to physical robot hardware in real-time.
The system supports distributed training and fine-tuning of large vision-language-action models across multiple GPUs and compute nodes. This includes the ability to adapt general base models to specific robot platforms using modular checkpoint adaptation and task-specific datasets.
Features
- Action Inference - Generates direct physical control commands for robots by processing visual and textual inputs through VLA models.
- Distributed Training - Spreads training workloads across multiple GPUs or nodes to reduce fine-tuning time for large models.
- Distributed Training Accelerators - Implements distributed GPU training to accelerate the fine-tuning of large vision-language-action models.
- Robotic Model Trainers - Functions as a distributed trainer for fine-tuning large vision-language-action models across multiple compute nodes.
- VLA - Adapts base vision-language-action model checkpoints to specific robot platforms using task-specific datasets.
- Robotics Frameworks - Provides a comprehensive framework for executing and fine-tuning vision-language-action models for robotic systems.
- Remote Action Streaming - Provides a websocket server for streaming model-generated control actions to physical robot hardware in real-time.
- Language and Robotics Integration - Generates physical control movements for robots by integrating vision and language inputs through pre-trained models.
- Robot Action Streaming Servers - Implements a websocket-based server that streams model-generated control actions to physical robots in real-time.
- Remote Inference Streaming - Uses persistent websocket connections to stream real-time robot control signals from remote inference servers.
- Remote Inference Streaming - Separates neural network computation from robot hardware by streaming inference to a dedicated remote server.
- Remote Hardware Controls - Sends real-time control actions to physical hardware via websocket connections from a remote inference server.
- Modular Checkpoint Adaptation - Enables the adaptation of general base models to specific robot platforms using modular checkpoint updates.
- Model Format Converters - Provides a tool for transforming robotic model checkpoints between different framework formats.
- Model Interoperability Formats - Converts model checkpoints between different framework formats to ensure compatibility across robotic environments.
- Weight Conversion Utilities - Provides utilities for transforming model checkpoints between different tensor formats for cross-environment compatibility.
- Distributed Learning - Supports training large robotic models across multiple GPUs and compute nodes to accelerate development.
- Embodied Foundation Models - Open-world generalist model for robotic control.
- Foundation Models - Flow-based vision-language-action models for robotic manipulation.
- Robotics Foundation Models - Flow-based vision-language-action model for general robot control.
