UI-TARS icon indicating copy to clipboard operation
UI-TARS copied to clipboard
verified publisher icon bytedance

Control iPhone?

Open boraturant opened this issue 7 months ago • 2 comments

I see in the codebase that there is android use case, but the "actions"(clicks, swipes..) on the real device are controled by ADB.

iPhone and Android devices can be projected onto desktop screen (iPhone Mirroring & scrcopy).

Can the mobile devices be controlled with UI-TARS-DESKTOP? (perhaps adding MOBILE-USE option?)

I think UI-TARS-DESKTOP or other app can process it from the desktop? Or the model is only finetuned with Android, not iPhone?

boraturant avatar May 07 '25 10:05 boraturant

Thanks for your feedback!

  1. The UI-TARS model itself works equally well for both iOS and Android interactions. 2.Due to Apple's closed ecosystem, we haven't built iOS operators yet (engineering challenge, not model limitation).
  2. As you've seen, we currently only handle Android devices via ADB in our CLI implementation.
  3. UI-TARS-Desktop is prioritizing computer/browser-use right now - we're actively matching industry trends and have big updates coming for these scenarios.
  4. Mobile-use is on our radar, but not before late Q2/early Q3.

We'll keep the community posted!

ZhaoHeh avatar May 07 '25 11:05 ZhaoHeh

thank you for your response. So techically I can do this by modifying the UI-TARS-Desktop code by

  • changing the prompt a bit
  • cropping the mirorred device screen from desktop and feed it into the model

boraturant avatar May 07 '25 13:05 boraturant