e2b-dev/open-computer-use
Open Computer Use
Open-computer-use is a framework designed to connect vision-capable language models to isolated cloud-based desktop environments. It functions as an agentic interface that enables autonomous systems to interact with graphical user interfaces by simulating mouse movements, keyboard keystrokes, and shell commands. By bridging language models with remote workspaces, the platform facilitates the execution of complex, long-running tasks within secure, sandboxed environments.
The platform distinguishes itself through its ability to orchestrate thousands of concurrent, isolated instances, making it suitable for large-scale AI evaluation and benchmarking. It maintains persistent session states, allowing for continuous workflows that require long-running background processes and consistent filesystem access. Users can integrate diverse vision and action models to customize agent behavior, while real-time visual streaming provides a feedback loop that allows agents to observe and respond to the desktop state.
Beyond core automation, the system supports secure data analysis and visualization by connecting external datasets to isolated environments. It offers flexible deployment options, including the ability to host sandbox infrastructure within private cloud accounts to meet specific data residency and security compliance requirements. The platform includes administrative tools for managing these environments, supported by user authentication to secure access to the orchestration dashboard.
Features
- Computer Use - Provides a toolkit for connecting vision-capable LLMs to isolated cloud desktop environments to automate complex tasks via mouse and keyboard.
- Desktop Automation - Provides an agentic interface for autonomous systems to interact with graphical user interfaces via simulated mouse, keyboard, and shell inputs.
- Desktop AI Agents - Enables autonomous AI agents to interact with graphical user interfaces by simulating mouse, keyboard, and visual feedback in remote environments.
- Cloud-Hosted Agent Sandboxes - Deploys and scales thousands of isolated execution environments to run code and evaluate AI agent performance.
- AI Model Integrations - Allows users to connect various vision, action, and grounding language models to the agent for flexible task execution.
- Vision-Language-Action Mappings - Processes visual snapshots of the desktop state to inform the next sequence of input actions taken by the language model.
- AI Observability and Evaluation - Launches thousands of concurrent, isolated sandbox instances to benchmark and test the performance of AI models across diverse computational tasks.
- Keyboard and Mouse Input Simulations - Translates high-level agent commands into low-level mouse movements and keyboard keystrokes to interact with standard graphical user interfaces.
- Rule Evaluation Scaling - Launches thousands of isolated sandbox instances simultaneously to run and evaluate multiple reward functions for reinforcement learning or large-scale research projects.
- Environment Orchestrators - Coordinates the lifecycle of thousands of concurrent isolated instances through a centralized control plane for scalable task execution.
- Private Cloud Deployments - Supports hosting sandbox infrastructure within personal or corporate cloud accounts to meet specific data residency and security compliance requirements.
- Code Execution Sandboxes - Supports running arbitrary code, terminal commands, and internet-enabled tasks within isolated and secure environments.
- Remote Desktop Infrastructure - Deploys and manages persistent, cloud-based virtual desktop environments that allow automated systems to perform complex, long-running workflows securely.
- Remote Desktop Protocols - Transmits real-time visual frames from a virtualized desktop to the agent while relaying input events back to the remote host.
- Virtualized Desktop Environments - Provides automated agents with persistent cloud-based desktop environments for complex computer-use tasks.
- Workflow Session Persistence - Maintains the filesystem and running processes of a remote environment across multiple interactions to support long-running automated workflows.
- Container-Based Sandboxes - Executes code and desktop environments within ephemeral, resource-constrained containers to ensure security and prevent host system contamination.
- Remote Desktop Environments - Enables automated agents to operate cloud-based desktop environments by simulating keyboard inputs, mouse movements, and shell commands.
- Agentic Desktop Interfaces - Provides a bridge between language models and remote graphical interfaces that enables autonomous navigation and interaction with desktop applications.
- Remote Desktop Monitoring - Streams real-time visual feedback from remote sandbox environments to allow agents to observe and respond to the desktop state.
Star history
How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
