Open-source alternatives to Agent S

This project is a computer control framework that uses multimodal vision models to simulate mouse and keyboard inputs for automating desktop tasks. It functions as an autonomous agent and vision-based orchestrator that interprets screen visuals to interact with user interfaces. The system employs vision language models and object detection to locate and click interface elements. It utilizes visual grounding to overlay numerical markers on UI components and uses optical character recognition to map on-screen text to precise pixel coordinates. The framework supports voice-controlled computing

Robotgo is a cross-platform desktop automation framework for the Go programming language. It provides a comprehensive toolkit for programmatically interacting with graphical user interfaces, enabling developers to simulate human input, manage application windows, and monitor system-wide hardware events. The library distinguishes itself through its low-level system integration, utilizing a foreign function interface to interact directly with native operating system APIs. It employs pixel-buffer memory mapping and real-time screen capture to perform visual element identification, allowing for i

PyAutoGUI is a Python GUI automation library and desktop automation framework. It provides a set of tools for programmatically controlling the mouse and keyboard to automate user interface interactions across different operating systems. The project functions as a cross-platform input simulator and computer vision screen scanner. It enables the simulation of keystrokes and cursor movements to perform repetitive tasks and utilizes screen analysis to locate specific images or pixel colors on the display. Its capability surface includes mouse and keyboard input simulation, screen image capture,

AutoKey is a Linux GUI macro manager and Python desktop automation tool. It functions as a background daemon that simulates keyboard and mouse events via the X11 protocol to automate interactions with graphical user interfaces. The software provides a Python scripting engine for executing stateful automation and a text expansion utility that replaces short abbreviations with predefined content or scripted actions in real time. It differentiates itself by combining low-level input simulation with the ability to manage global state and persistent variables across different scripts. The project

Cua is an agent benchmarking and desktop automation platform designed to evaluate autonomous agents and execute repetitive tasks within isolated, virtualized environments. It provides a framework for provisioning consistent workspaces and measuring agent performance against standardized desktop operations. The platform distinguishes itself by integrating virtual machine orchestration with headless interaction capabilities. By leveraging hypervisor-based virtualization, it runs operating systems at near-native speeds, while its automation layer injects commands directly into application proces

OmniParser is a multimodal interaction engine designed to function as a desktop automation agent. It interprets visual screen information to execute complex, multi-step tasks across operating system environments by bridging visual interface perception with language models. Through a continuous cycle of observation and command execution, the system grounds high-level natural language instructions into precise, coordinate-based actions. The project distinguishes itself by utilizing vision-based parsing to interact with software interfaces without requiring access to underlying application progr

xdotool is a command-line utility for X11 desktop automation that simulates keyboard and mouse input and manages application windows. It functions as a scriptable tool that maps sequential text instructions to low-level library calls to control the X11 graphical environment. The project enables the generation of synthetic keyboard and mouse events to automate interactions with graphical user interfaces. It also acts as a window manager controller, allowing users to programmatically move, resize, focus, and organize application windows and virtual desktops. The tool covers a wide range of cap

UI-TARS-desktop is a cross-platform desktop application designed to automate software interface interactions. It functions as a local agent environment that interprets graphical user interfaces through multimodal visual-language model reasoning, allowing it to navigate and manipulate software by simulating human-like mouse and keyboard inputs. The platform distinguishes itself by executing all visual recognition and decision-making logic directly on the host machine. This local inference model ensures that screen data and sensitive information remain private, as no processing is offloaded to

Glazewm is a tiling window manager and desktop automation framework designed to organize application windows into non-overlapping layouts. It functions as a keyboard-driven controller, allowing users to manage window focus, workspace navigation, and layout configurations entirely through custom keybindings. By intercepting system-level events, the software overrides default window placement and behavior to maximize screen space and maintain a structured workspace. The project distinguishes itself through a highly configurable environment that supports complex multi-monitor setups and dynamic

SerpentAI is a game AI development kit and computer vision framework designed for building autonomous agents that interact with video games. It serves as a game input automation tool and a machine learning model integration engine, allowing developers to create agents that perceive game states and execute actions. The framework utilizes a plugin-based agent architecture to provide modular extensions for game-specific logic and behaviors. It features a specialized system for training, bundling, and deploying machine learning classifiers to recognize visual contexts and game states in real time

This project is a Python framework for building autonomous, event-driven agent systems. It provides a unified runtime for orchestrating multi-agent workflows, managing persistent conversation state, and executing code within secure, isolated sandbox environments. The framework is designed to handle complex task delegation, allowing agents to invoke other agents as tools while maintaining context across multi-turn interactions. The framework distinguishes itself through its deep integration with the Model Context Protocol, enabling agents to connect to external data sources and remote services

Maestro is a declarative mobile and web UI automation framework designed for end-to-end testing. It operates by querying the native accessibility tree of an application, allowing for black-box testing without requiring source code instrumentation or platform-specific dependencies. The framework distinguishes itself through a unified command syntax that abstracts interactions across Android, iOS, and web environments. It features a dynamic synchronization engine that automatically pauses test execution to account for non-deterministic animations and network-dependent content loading, ensuring

Kap is an open-source screen recording application designed to capture desktop activity as video or image files. It provides a comprehensive workflow for recording screen sessions, including options for capturing system audio and highlighting mouse clicks to assist in the creation of instructional content. The application distinguishes itself through a modular, plugin-based architecture that allows for the integration of external services for sharing and uploading media. It utilizes a background transcoding utility to process and convert captured recordings into various formats, such as GIF,

Tambo is an orchestration platform and framework designed for building generative user interfaces and conversational AI agents. It provides the infrastructure to manage persistent chat threads, execute multi-step reasoning workflows, and integrate large language models with external tools and services. By combining an agent orchestration layer with a component-based library, the project enables developers to create interactive interfaces where AI models dynamically render and update UI elements in real-time. The framework distinguishes itself through its generative UI capabilities, which allo

OSWorld is an evaluation framework and multimodal agent benchmark designed to test the ability of large language models to complete complex tasks within virtualized operating system environments. It provides a virtualized desktop sandbox and a virtual machine orchestrator to deploy, snapshot, and reset cloud-based desktops, ensuring reproducible test states for AI agent interactions. The system distinguishes itself by providing an OS-level action space that translates model decisions into mouse clicks, keyboard inputs, and system commands. It employs a standardized interface to integrate vari

gptme is an autonomous AI agent server and framework designed for local system automation, software development, and code execution. It operates as a local execution engine that enables language models to run shell commands, modify local files, and interact with the operating system. The project functions as a Model Context Protocol client, integrating with external servers to expand agent capabilities with standardized tools and data sources. It features a provider-agnostic routing system to orchestrate tasks across multiple proprietary cloud APIs and local AI backends. The system includes

Kooha is a screen recorder for Linux desktops that utilizes the Wayland protocol and XDG Portals for secure recording. It functions as a hardware-accelerated screen capture tool that offloads video compression to the GPU to reduce CPU load and power consumption. The application integrates the PipeWire framework to capture system and microphone audio streams and leverages FFmpeg for muxing video streams and exporting various codecs and containers. Its user interface is a native Linux application built with the GTK toolkit. The software covers screen recording and capture of entire displays, s

LangChainJS is an AI agent orchestrator and application framework designed for building autonomous systems that use large language models to plan and execute tasks. It serves as an integration library that connects language models with tools, memory, and external data sources to create context-aware logic and complex workflows. The project provides a provider-agnostic interface and model provider abstraction, allowing applications to switch between different language model providers without rewriting core logic. It includes a toolkit for retrieval augmented generation, utilizing retrievers to

Osaurus is a local AI workflow engine and LLM agent orchestration framework designed for private execution on local hardware. It functions as a desktop application automator and a voice-controlled AI interface, enabling the development of autonomous agents that can write code, execute tools, and operate a computer without keyboard or mouse input. The system is distinguished by its ability to control native desktop applications via accessibility APIs and manage web interactions through a headless browser automation tool. It supports a local-first execution model and on-premises deployment with

DeepSeek-TUI is an AI coding agent orchestrator and framework designed to automate complex programming tasks. It functions as a harness for coordinating AI models that can read source code, edit files, and execute shell commands through automated agent workflows. The system is distinguished by its multi-agent coordination capabilities, which allow for the spawning of parallel sub-agents to handle concurrent investigations or implementation slices. It employs autonomous goal-seeking loops to pursue objectives across multiple turns and utilizes a tool integration gateway to connect models to ex

Hermes-agent is an autonomous AI agent framework and runtime designed to execute complex tasks and synthesize new skills from execution traces. It includes a provider-agnostic gateway for routing requests across multiple model backends and a serverless runtime that suspends idle agent instances and resumes them on demand across containers and virtual machines. The project provides a desktop automation toolset that controls native GUI workflows on Linux by querying accessibility APIs and injecting input events. It further distinguishes itself with the ability to generate procedural skills from

Yabai is a system-level utility for macOS that provides automated window management and workspace organization. It functions as a tiling window manager that uses binary space partitioning to arrange application windows into grid layouts, bypassing standard desktop constraints through direct interaction with the system window server. The project distinguishes itself by offering granular control over the graphical user interface via a socket-based command interface and event-driven system hooks. This allows for the programmatic manipulation of window states, display configurations, and virtual

Handy is a local speech-to-text automation tool designed to convert spoken audio into text and inject it directly into active desktop applications. By running machine learning models entirely on the host hardware, it provides a private, offline-first environment for dictation and command execution. The system functions as a background service that manages microphone input, transcription state, and text output, enabling hands-free typing across various software environments. The project distinguishes itself through a modular pipeline that integrates local language models for post-transcription

UI-TARS is an LLM GUI automation framework and multimodal action grounding system. It functions as a GUI agent orchestrator and cross-platform device controller that uses large language models to interpret graphical interfaces and execute actions across desktop and mobile operating systems. The system translates model-generated coordinates into precise screen positions to interact with visual user interface elements. It employs a multimodal approach to interpret screen layouts and decomposes complex goals into multi-step trajectories through reasoning and error correction. The project provid

ShareX is a desktop utility designed for screen capture, image annotation, and automated file sharing. It provides a comprehensive suite of tools for capturing screen regions, windows, or scrolling content, and includes a layered image editor that allows users to manipulate, scale, and transform graphical elements and annotations directly on captured media. The application distinguishes itself through an event-driven post-capture pipeline that triggers automated workflows, such as image processing, external command execution, or file uploads, immediately after a capture event. Users can exten

ScreenToGif is a screen recording and GIF editor designed to capture screen activity and refine frames to produce GIF animations and video files. It functions as a screen-to-video capture tool for creating visual demonstrations, documentation, and tutorials. The application includes a non-linear frame editor that allows for the modification of recorded footage before export. It can capture a selected screen area, webcam feed, or sketchboard, and exports the resulting animations as images, videos, or GIFs. The tool supports user interface translation to provide a localized experience in multi

cc-haha is a cross-platform desktop agent and computer use framework that enables large language models to control local operating systems through screenshots, clicks, and keystrokes. It functions as an AI coding workbench and orchestration platform, allowing for the management of multi-project workflows and the coordination of multiple agents executing complex tasks in parallel. The system includes a model backend gateway to connect various artificial intelligence providers and local models to autonomous agents. It features a centralized permission gate for authorizing sensitive commands and