2 dépôts
Awesome GitHub RepositoriesInterface Grounding
The process of mapping high-level instructions to specific coordinates and components on a device interface.
Distinct from Hardware Device Interfaces: Different from hardware interfaces or network ground systems; this is AI-driven UI coordinate mapping.
Explore 2 awesome GitHub repositories matching artificial intelligence & ml · Interface Grounding. Refine with filters or upvote what's useful.
Awesome Interface Grounding GitHub Repositories
Trouvez les meilleurs dépôts grâce à l'IA.Nous recherchons les dépôts les plus pertinents grâce à l'IA.
- apple/ml-ferret
apple/ml-ferret
8,680Voir sur GitHubml-ferret is a multimodal large language model framework and visual reasoning engine designed to reason about images and user interfaces. It functions as a UI grounding model and referring expression comprehension tool that maps natural language descriptions to precise pixel coordinates. The system focuses on high-resolution image analysis to identify and locate specific interface components. It employs multi-resolution image processing and region-aware visual encoding to preserve detail across different aspect ratios, enabling the model to analyze spatial relationships and functional layouts
Maps natural language descriptions to precise pixel coordinates to identify and locate interface components.
Python - x-plug/mobileagent
X-PLUG/MobileAgent
7,218Voir sur GitHubMobileAgent is an LLM-powered mobile automation agent and framework designed to navigate mobile user interfaces and execute multi-step tasks. It functions as a device interface automation system that maps semantic commands to screen coordinates to perform input events across mobile operating systems. The project operates as a cross-app workflow orchestrator, switching between native on-screen interface actions and external API tools to complete sophisticated operations. It includes a visual grounding system that analyzes screenshots and interface metadata to identify elements and validate the
Maps high-level instructions to specific screen coordinates and interface components for mobile interaction.
Pythonagentandroidapp
