Microsoft OmniParser

Microsoft OmniParser is an open-source, vision-based screen parsing module designed to bridge the gap between graphical user interfaces (GUIs) and large language models. It converts standard UI screenshots into structured, machine-readable elements (identifying interactive regions, bounding boxes, and functional descriptions) without requiring access to underlying source code like HTML or Android view hierarchies. By translating visual interfaces into clear semantic data, it significantly enhances the capability of vision-based AI agents and models (such as GPT-4V, Claude, and DeepSeek) to accurately interact with desktop, web, and mobile environments.