awesome-repositories.com
Blog
awesome-repositories.com

Descubre los mejores repositorios open-source con nuestra búsqueda potenciada por IA.

ExplorarBúsquedas curadasAlternativas open-sourceSoftware autohospedableBlogMapa del sitio
ProyectoAcerca deCómo clasificamosPrensaServidor MCP
Aviso legalPrivacidadTérminos
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
·
simular-ai avatar

simular-ai/Agent-S

0
View on GitHub↗
11,855 estrellas·1,401 forks·Python·Apache-2.0·11 vistaswww.simular.ai↗

Agent S

Agent-S is a multimodal AI agent and LLM desktop automation framework designed to control operating systems through graphical user interface interactions. It functions as a computer use interface, utilizing vision-language grounding to translate natural language goals into precise screen coordinates and system actions.

The project differentiates itself by combining structured accessibility tree inspection with vision-based element localization. It manages cross-application workflows by mapping conceptual descriptions to physical pixels and simulating low-level keyboard and mouse events to move data between disparate software.

Its broader capabilities cover hierarchical task planning, multimodal state observation, and native code execution for problem solving. The system also includes comprehensive media handling for screen capture and audio transcription, filesystem management, and interaction error recovery to refine task outcomes.

The framework provides a command-line interface for executing standalone automation scripts without a separate build step.

Features

  • AI Agent Orchestrators - Integrates multimodal models to autonomously execute complex goals described in natural language.
  • GUI and Computer Agents - Performs high-level interactions on interface components, including clicking buttons and setting text.
  • Desktop Automation - Controls mouse and keyboard inputs to automate complex tasks across various OS applications.
  • Model Provider Integrations - Implements unified interfaces to connect and configure various external and local language model providers.
  • Computer Vision Localization - Translates conceptual descriptions of UI elements into precise pixel coordinates on screenshots.

Búsqueda con IA

Explora más repositorios increíbles

Describe lo que necesitas en lenguaje sencillo: la IA clasifica miles de proyectos open-source curados por relevancia.

Start searching with AI
  • Cross-Application Workflow Automation - Automates multi-step sequences to move data between disparate desktop apps and web browsers.
  • Desktop Automation Frameworks - Uses large language models to control the OS by combining accessibility tree data and visual analysis.
  • Language Model Querying - Provides the core capability to send text and image prompts to LLMs to retrieve reasoning and responses.
  • AI Model Integrations - Provides adapters and interfaces to connect multimodal models including chat, visual grounding, and speech-to-text.
  • OCR Engines - Uses an OCR engine to identify and locate bounding boxes of text within the user interface.
  • Task Planning Systems - Decomposes high-level goals into manageable subtasks using hierarchical planning and real-time knowledge.
  • Vision-Language Grounding Models - Utilizes models that map natural language instructions to specific spatial coordinates on a visual user interface.
  • Multimodal Agents - Processes screenshots and accessibility data using vision and language models to execute complex tasks.
  • Natural Language Automation - Executes sequences of primitive actions to complete goals described in plain text.
  • Application Lifecycle Managers - Locates and launches software applications and navigates to web URLs within the desktop environment.
  • Desktop Applications - Manages the launching and navigation of desktop applications and web addresses.
  • Keyboard Input Automation - Sends text, key commands, and raw physical key presses directly to the operating system.
  • Mouse Control Automation - Simulates mouse button events to trigger clicks at specific screen coordinates across different OSs.
  • Keyboard Shortcuts - Executes specific key combinations with modifiers to trigger application-level commands.
  • Screen Capture Tools - Captures full-screen snapshots of the desktop, including an option to hide the cursor.
  • Screen Capture Utilities - Captures the graphical user interface as base64 strings to enable visual perception for the AI agent.
  • Focus Navigation Controllers - Manages window focus and visibility transitions when switching between different applications.
  • Region Capture - Takes screenshots of specific coordinate-defined regions to isolate particular UI areas for analysis.
  • Accessibility Tree Accessors - Deno GUI Automation retrieves the root accessibility node of the currently focused application or a specific process ID.
  • Desktop Application Automation - Provides the ability to identify and retrieve handles for specific desktop applications and the default browser.
  • Accessibility Tree Automation - Activates the accessibility tree of focused applications to enable programmatic inspection and interaction.
  • GUI Element Localizations - Deno GUI Automation finds specific interface nodes using direct child traversal or natural-language concept searches.
  • GUI Structure Inspections - Deno GUI Automation captures a snapshot of a window's accessibility tree to identify interactable components.
  • Window Context Binding - Connects the agent to a target interface using foreground window handles, process IDs, or platform identifiers.
  • Multimodal Desktop Observers - Implements a multimodal state observation system combining accessibility snapshots and image data.
  • Desktop Application Workflows - Executes multi-step sequences across different software to move data between disparate applications.
  • Accessibility-Tree-Based Locators - Traverses the system accessibility layer to locate interactive UI elements via semantic roles and names.
  • User Interaction Simulation - Simulates human-like keystrokes and mouse events using a cartesian coordinate system.
  • Accessibility Role Mapping - Uses standardized accessibility roles to consistently discover and identify specific interface components.
  • Window Visibility Controllers - Determines whether launched applications remain visible or hidden to manage the user interface state.
  • Coordinate Converters - Translates relative screen coordinates into global physical pixels to ensure precise GUI interaction.
  • Semantic - Locates elements within the accessibility tree using natural-language queries and similarity scoring.
  • Visual Analysis Processors - Captures structural text and imagery of the screen for processing by vision models.
  • Visual Localization Tools - Identifies physical pixel coordinates of visual concepts within screenshots using grounding models.
  • Process-Level Input Injectors - Simulates low-level keyboard and mouse events to interact with graphical user interfaces at the OS level.
  • UI Element Selectors - Searches for interface elements using a combination of ARIA roles, traversal orders, and name filters.
  • UI Inspection Tools - Retrieves global physical bounding boxes of UI elements and ensures they are scrolled into view.
  • Interaction Area Definitions - Specifies rectangular regions on the screen using coordinates to identify elements for agent interaction.
  • Window Management - Adjusts the size and visibility of on-screen windows through minimize, maximize, and close actions.
  • Window State Controls - Modifies window states such as minimizing, maximizing, or closing application instances.
  • Window Lifecycle Controllers - Controls top-level window states including minimizing, maximizing, and closing.
  • Agent Capability Extensions - Provides mechanisms to integrate custom tools and capabilities to extend the agent's behavior for specific use cases.
  • Audio Transcription - Converts captured audio chunks into text transcripts using integrated speech-to-text models.
  • Code Execution Agents - Generates and executes native code directly to solve complex problems through system interactions.
  • Coordinate Normalization Utilities - Determines whether screen coordinates are absolute pixel positions or relative percentages for visual grounding.
  • Dynamic Plan Refinement - Adjusts subgoals and action sequences based on environmental feedback and new observations.
  • Knowledge and Memory - Stores and retrieves successful task trajectories to refine and optimize future automation actions.
  • Visual State Verifications - Evaluates whether the current screen satisfies specific conditions or contains required interface elements.
  • Screen Capture Extraction - Captures screen regions and uses language models to parse visual content into structured data.
  • Metadata Extraction - Retrieves specific metadata from a UI element, such as its role, title, or current value.
  • Accessibility Tree Exporting - Renders a window's accessibility subtree as a structured text representation for AI analysis.
  • System Clipboard Access - Reads and writes data to the system clipboard to facilitate information transfer between disparate applications.
  • Display Automation Tools - Retrieves pixel dimensions and coordinates of physical screens to determine the available workspace area.
  • Execution Sampling Strategies - Optimizes task success rates by running multiple attempts in parallel and sampling the best outcome.
  • Web Search Integrations - Integrates external search engine APIs to retrieve real-time web information for task context.
  • Image Annotation Tools - Draws reference grids over images to provide spatial coordinates for visual grounding.
  • Image Processing - Performs pixel-level transformations including resizing, compression, and grid overlays for visual analysis.
  • Image Compression Tools - Compresses images into JPEG format with configurable quality settings to optimize size.
  • Base64 Image Decoders - Encodes images to base64 data URLs and decodes base64 strings back into image objects.
  • Image Transformation Utilities - Resizes images to fit within specified bounds while maintaining the original aspect ratio.
  • Cursor State Retrieval - Returns the current x and y coordinates of the mouse cursor in pixels.
  • Window Handle Management - Provides the ability to retrieve handles for all visible top-level windows or specific processes.
  • Error Recovery - Detects failures in GUI actions and self-corrects by adapting navigation or grounding methods.
  • Success-Based Parallel Sampling - Executes multiple task attempts in parallel to select the most successful outcome.
  • Execution Logs - Records and streams workflow-level execution history to facilitate real-time monitoring and debugging.
  • Accessibility Inspection Tools - Analyzes the system accessibility layer to identify UI components by roles and structure.
  • Focus Management - Moves keyboard focus to specific elements and manages foreground window state.
  • Drag and Drop Simulations - Enables moving objects and data between different desktop applications through simulated drag-and-drop actions.
  • Programmatic Scrolling - Implements programmatic vertical and horizontal scrolling using pixel deltas to navigate user interfaces.
  • Agent Frameworks - Agentic framework for human-like computer interaction.
  • Agent Memory Systems - Open agentic framework for computer control using human-like interaction.
  • AI Agents - GUI agent capable of multi-app collaboration and self-learning.
  • AI Agents and Automation - Open agentic framework that autonomously interacts with computer GUIs like a human.
  • Computer Use - Framework for human-like computer interaction.

    • Historial de estrellas

    Gráfico del historial de estrellas de simular-ai/agent-sGráfico del historial de estrellas de simular-ai/agent-s

    Alternativas open-source a Agent S

    Proyectos open-source similares, clasificados según cuántas características comparten con Agent S.
    • bytebot-ai/bytebotAvatar de bytebot-ai

      bytebot-ai/bytebot

      10,413Ver en GitHub↗

      Bytebot is an LLM desktop automation framework and virtual Linux desktop environment. It enables AI agents to plan and execute mouse and keyboard actions on a virtual computer using natural language, allowing for autonomous desktop automation and the integration of legacy systems that lack native APIs. The system operates as an LLM API gateway and a Model Context Protocol server, routing requests across multiple language model providers with integrated load balancing and rate limiting. It provides isolated, containerized environments where agents use visual reasoning to interpret screenshots

      TypeScriptagentagentic-aiagents
      Ver en GitHub↗10,413
    • hammerspoon/hammerspoonAvatar de Hammerspoon

      Hammerspoon/hammerspoon

      14,497Ver en GitHub↗

      Hammerspoon is a programmable automation engine for macOS that enables deep system-level control through a Lua scripting environment. By bridging high-level scripts with native Objective-C APIs, it allows users to interact with the operating system's accessibility tree, intercept hardware input streams, and manage the lifecycle of running applications. The project distinguishes itself through an event-driven architecture that registers asynchronous hooks for system notifications and hardware events. This allows for real-time automation, such as remapping keyboard and mouse inputs, managing wi

      Objective-Cautomationhammerspoonirc
      Ver en GitHub↗14,497
    • octalmage/robotjsAvatar de octalmage

      octalmage/robotjs

      12,740Ver en GitHub↗

      Robotjs is a native Node.js automation library and desktop input simulator. It uses C++ bindings to provide low-level access to operating system functions, allowing for the programmatic control of the mouse and keyboard and the analysis of screen pixels. The library functions as a toolkit for automating user interfaces and desktop workflows, including those within Electron applications. It enables the simulation of key presses and mouse movements to automate interactions with desktop software and perform automated data entry. Its capabilities extend to screen pixel analysis, where it capture

      C
      Ver en GitHub↗12,740
    • othersideai/self-operating-computerAvatar de OthersideAI

      OthersideAI/self-operating-computer

      10,153Ver en GitHub↗

      This project is a computer control framework that uses multimodal vision models to simulate mouse and keyboard inputs for automating desktop tasks. It functions as an autonomous agent and vision-based orchestrator that interprets screen visuals to interact with user interfaces. The system employs vision language models and object detection to locate and click interface elements. It utilizes visual grounding to overlay numerical markers on UI components and uses optical character recognition to map on-screen text to precise pixel coordinates. The framework supports voice-controlled computing

      Pythonautomationopenaipyautogui
      Ver en GitHub↗10,153
    Ver las 30 alternativas a Agent S→

    Preguntas frecuentes

    ¿Qué hace simular-ai/agent-s?

    Agent-S is a multimodal AI agent and LLM desktop automation framework designed to control operating systems through graphical user interface interactions. It functions as a computer use interface, utilizing vision-language grounding to translate natural language goals into precise screen coordinates and system actions.

    ¿Cuáles son las características principales de simular-ai/agent-s?

    Las características principales de simular-ai/agent-s son: AI Agent Orchestrators, GUI and Computer Agents, Desktop Automation, Model Provider Integrations, Computer Vision Localization, Cross-Application Workflow Automation, Desktop Automation Frameworks, Language Model Querying.

    ¿Qué alternativas de código abierto existen para simular-ai/agent-s?

    Las alternativas de código abierto para simular-ai/agent-s incluyen: bytebot-ai/bytebot — Bytebot is an LLM desktop automation framework and virtual Linux desktop environment. It enables AI agents to plan and… hammerspoon/hammerspoon — Hammerspoon is a programmable automation engine for macOS that enables deep system-level control through a Lua… octalmage/robotjs — Robotjs is a native Node.js automation library and desktop input simulator. It uses C++ bindings to provide low-level… othersideai/self-operating-computer — This project is a computer control framework that uses multimodal vision models to simulate mouse and keyboard inputs… go-vgo/robotgo — Robotgo is a cross-platform desktop automation framework for the Go programming language. It provides a comprehensive… asweigart/pyautogui — PyAutoGUI is a Python GUI automation library and desktop automation framework. It provides a set of tools for…