Inquiry about the Accuracy of Visual Perception Module in Mobile-Agent-v2

[XinrunXu]

Hi,

I am currently reading your paper on Mobile-Agent-v2 and am impressed with the multi-agent architecture proposed.

While going through the paper, I noticed the detailed explanation of the visual perception module's design, including the use of the text recognition tool (ConvNextViT-document), icon recognition tool (GroundingDINO), and icon description tool (Qwen-VL-Int44).

However, I couldn't find any specific data regarding the accuracy of this module in recognizing, localizing, and describing UI elements.

Could you please provide some insights into the performance of the visual perception module?

Thank you for your time and valuable insights!

[junyangwang0410]

Hello. Since Mobile-Agent-v2 is the first framework to use OCR, SAM and VLLM for mobile icon detection, screen text recognition and mobile icon description, we cannot directly obtain performance from the conclusions of existing work. We have not done specific experiments to quantify the effects of these models. However, from our experience, the current performance of OCR and VLLM is overflowing. On the contrary, the performance of SAM for icon detection is still insufficient. To address this, the solution we adopted is oversampling, that is, using a lower confidence threshold to obtain a large number of icons, and the decision agent will do the screening in the final decision.