1 dépôt
Awesome GitHub RepositoriesVisual UI Labeling
Overlays numerical markers or labels on detected user interface components for AI referencing.
Distinct from Object Detection: Focuses on adding referenceable markers to UI elements rather than just detecting their existence
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Visual UI Labeling. Refine with filters or upvote what's useful.
Awesome Visual UI Labeling GitHub Repositories
Trouvez les meilleurs dépôts grâce à l'IA.Nous recherchons les dépôts les plus pertinents grâce à l'IA.
- othersideai/self-operating-computer
OthersideAI/self-operating-computer
10,153Voir sur GitHubThis project is a computer control framework that uses multimodal vision models to simulate mouse and keyboard inputs for automating desktop tasks. It functions as an autonomous agent and vision-based orchestrator that interprets screen visuals to interact with user interfaces. The system employs vision language models and object detection to locate and click interface elements. It utilizes visual grounding to overlay numerical markers on UI components and uses optical character recognition to map on-screen text to precise pixel coordinates. The framework supports voice-controlled computing
Overlays numerical markers on detected UI components to help the AI reference specific elements by ID.
Pythonautomationopenaipyautogui
