[Bug Report] Why my Apple vision pro cannot properly render Isaac Lab environment?

[DAIJinJ]

Describe the bug

I'm trying to teleoperate an Isaac Lab robot with apple vision pro. While in AR mode, the teleoperation data can be properly received (Robotic arms and hands in Isaac Lab move according to tracking data published by apple vision pro), but the virtual graphical environment cannot be properly rendered on apple vision pro (only displaying colorful mosaics/pixels). Steps to reproduce

I followed Option 1 in isaac-sim.github.io/IsaacLab/main/source/how-to/cloudxr_teleoperation.html#known-issues both Isaac Lab and CloudXR containers together using Docker Compose, and built and ran the Isaac XR Teleop Sample Client on Apple Vision Pro.

1. ./docker/container.py start \ --files docker-compose.cloudxr-runtime.patch.yaml \ --env-file .env.cloudxr-runtime
2. ./docker/container.py enter base Then，I run an example teleop task with: 3../isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \ --task Isaac-PickPlace-GR1T2-Abs-v0 \ --teleop_device handtracking \ --enable_pinocchio

After clicking "Start AR" in the Isaac Sim UI, I received the following error:

`2025-10-11T02:42:23Z__ [875,296ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION (secondary thread): in TfPyConvertPythonExceptionToTfErrors at line 114 of /builds/omniverse/usd-ci/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

sys:1: RuntimeWarning: coroutine 'ScriptManager._pending_sync_task_handler' was never awaited RuntimeWarning: Enable tracemalloc to get the object allocation traceback 2025-10-11T02:42:23Z [875,303ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION (secondary thread): in TfPyConvertPythonExceptionToTfErrors at line 114 of /builds/omniverse/usd-ci/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2025-10-11T02:42:23Z [875,306ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION (secondary thread): in TfPyConvertPythonExceptionToTfErrors at line 114 of /builds/omniverse/usd-ci/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2025-10-11T02:42:23Z [875,309ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION (secondary thread): in TfPyConvertPythonExceptionToTfErrors at line 114 of /builds/omniverse/usd-ci/USD/pxr/base/tf/pyError.cpp -- Tf Python`

System Info

Isaac Sim Version: [5.0] OS: [Ubuntu 22.04] GPU: [ 2 x RTX 3090] CUDA: [12.9] GPU Driver: [575.64.03 ]

[DAIJinJ]

After I rebuilt the Docker container, although the terminal no longer shows errors, the Apple Vision Pro still displays a mosaic pattern.

[DAIJinJ]

The terminal shows `XR_ERROR_VALIDATION_FAILURE: xrWaitFrame(frameState->type == 0) LOG in xrCreateInstance: Instance created createInfo->applicationInfo.applicationName: Isaac-Sim createInfo->applicationInfo.applicationVersion: 1 createInfo->applicationInfo.engineName: Omniverse Kit XR createInfo->applicationInfo.engineVersion: 107039000 createInfo->applicationInfo.apiVersion: 1.1.45 appinfo.detected.engine.name: (null) appinfo.detected.engine.version: 0.0.0 quirks.disable_vulkan_format_depth: false quirks.disable_vulkan_format_depth_stencil: false quirks.no_validation_error_in_create_ref_space: true quirks.skip_end_session: false quirks.parallel_views: false

LOG in xrGetSystem: Selected devices Head: 'Head Device' Eyes: '' Left: 'Controller Left' Right: 'Controller Right' Gamepad: '' Hand-Tracking Left: 'Head Device' Hand-Tracking Right: 'Head Device' XR_ERROR_FORM_FACTOR_UNSUPPORTED: xrGetSystem(getInfo->formFactor) no matching system (given: 2, first: 1) LOG in xrCreateVulkanInstanceKHR: Creation of VkInstance: result: VK_SUCCESS vulkanInstance: 0x71cc1c9c8e40 extensions: VK_KHR_external_fence_capabilities VK_KHR_external_memory_capabilities VK_KHR_external_semaphore_capabilities VK_KHR_get_physical_device_properties2 VK_EXT_debug_utils LOG in xrGetVulkanGraphicsDeviceKHR: Creation of VkDevice: result: VK_SUCCESS vulkanDevice: 0x71dabf0da170 vulkanInstance: 0x71cc1c9c8e40 external_fence_fd: true external_semaphore_fd: true timelineSemaphore: true extensions: VK_KHR_dedicated_allocation VK_KHR_external_fence VK_KHR_external_memory VK_KHR_external_semaphore VK_KHR_get_memory_requirements2 VK_KHR_external_memory_fd VK_EXT_depth_range_unrestricted VK_KHR_external_semaphore_fd VK_KHR_external_fence_fd VK_KHR_image_format_list VK_KHR_timeline_semaphore 2025-10-13T01:04:52Z [146,620,909ms] [Warning] [omni.usd] Coding Error: in AddXformOp at line 189 of /builds/omniverse/usd-ci/USD/pxr/usd/usdGeom/xformable.cpp -- The xformOp 'xformOp:transform' already exists in xformOpOrder [[xformOp:transform]].`

[RandomOakForest]

Thank you for posting this, the team will investigate this further.

[Joao-G-Martins]

Hi!

Is there any news on this?

I encountered the same issue using a Meta Quest 3, although the Meta Quest 3 is not officially supported by IsaacLab. However, cloudXR has provided an Android sample that allows it to detect both the head and controllers via openXR. Nevertheless, when connecting to the server, IsaacLab also receives the same error: XR_ERROR_FORM_FACTOR_UNSUPPORTED.

Thanks!