arfoundation-samples
arfoundation-samples copied to clipboard
Unity-Technologies
When running on iOS 16 the camera feed is unstable (shaky and jittery) when multi-threaded rendering is enabled
Unity bug report case number CASE IN-7240
Describe the bug When running on iOS 16 the camera feed is unstable (shaky and jittery) when multi-threaded rendering is enabled
To Reproduce Steps to reproduce the behavior: Enable multi-threaded rendering and build this samples repo branch 4.1 to an iPhone 12 Pro Max running iOS 16. Open the simple AR scene and start scanning around to generate a few planes, if the camera feed is still stable at this point, lower the application and reopen it. Then the camera feed will go bananas.
Expected behavior I would expect the camera feed to be stable and usable like previous versions of iOS.
Actual behavior The camera feed is jumpy and jittery, also when in this state pressing the return button will crash the app.
Smartphone (please complete the following information):
- Device: iPhone 12 Pro Max, also reproduced with iPhone 11 and iPad Pro M1 however they seemed to require more geometry to get into this strange state
- OS: iOS 16
- Unity version: 2020.3.35f1
- ARFoundation version: 4.1.10
We were able to reproduce the bug internally. You can follow the progress on it at the public issue tracker. https://issuetracker.unity3d.com/issues/ar-camera-feed-jitters-and-shakes-with-mutithreaded-rendering-when-running-on-ios-16
Hello, do you have any update to share on this matter?
iOS 16 will probably be released within a month, which means developers will have a short amount of time to update and ship their app with the ARKit plugin update that will fix this issue.
Update: We have a fix for this issue, which will be released in the new patched versions of the packages: 4.1.11, 4.2.5, 5.0.1, as soon as they pass QA validation. https://forum.unity.com/threads/unity-arkit-support-issue-with-ios-16.1334162/
Update: We have released new versions of the packages with a fix for the issue that causes jittering of background texture on iOS 16.
- 5.0.2
- 4.2.6
- 4.1.12
I tested the solution in Unity 2021.1.19f on iPhoneX and 12 Pro Max after updating the package to version 4.1.12 the background jittering stopped. But I did note some random app crashes - at the moment I can't say what might be causing them, however Crashlytics noted the fallowing entries.
Crashed: com.Metal.CompletionQueueDispatch 0 CoreFoundation 0x7ba2c CFRelease + 28 1 UnityFramework 0xd63ec4 ___ZN12_GLOBAL__N_121ARKitXRCameraProvider31ScheduleReleaseRetainedTexturesEv_block_invoke + 24 2 Metal 0x20dac MTLDispatchListApply + 44 3 Metal 0xc55c -[_MTLCommandBuffer didCompleteWithStartTime:endTime:error:] + 516 4 IOGPU 0x51b8 -[IOGPUMetalCommandBuffer didCompleteWithStartTime:endTime:error:] + 216 5 Metal 0x20e3c -[_MTLCommandQueue commandBufferDidComplete:startTime:completionTime:error:] + 104 6 IOGPU 0x1192c IOGPUNotificationQueueDispatchAvailableCompletionNotifications + 120 7 IOGPU 0x5f64 __IOGPUNotificationQueueSetDispatchQueue_block_invoke + 60 8 libdispatch.dylib 0x6487c _dispatch_client_callout4 + 16 9 libdispatch.dylib 0x377c0 _dispatch_mach_msg_invoke$VARIANT$armv81 + 372 10 libdispatch.dylib 0x3f72c _dispatch_lane_serial_drain$VARIANT$armv81 + 308 11 libdispatch.dylib 0x3835c _dispatch_mach_invoke$VARIANT$armv81 + 444 12 libdispatch.dylib 0x3f72c _dispatch_lane_serial_drain$VARIANT$armv81 + 308 13 libdispatch.dylib 0x40318 _dispatch_lane_invoke$VARIANT$armv81 + 432 14 libdispatch.dylib 0x3f72c _dispatch_lane_serial_drain$VARIANT$armv81 + 308 15 libdispatch.dylib 0x402e4 _dispatch_lane_invoke$VARIANT$armv81 + 380 16 libdispatch.dylib 0x4a000 _dispatch_workloop_worker_thread + 612 17 libsystem_pthread.dylib 0x1b50 _pthread_wqthread + 284 18 libsystem_pthread.dylib 0x167c start_wqthread + 8
And the second occurring more frequently:
Crashed: com.Metal.CompletionQueueDispatch 0 CoreFoundation 0x80d44 CFRelease + 88 1 UnityFramework 0xd642d4 invocation function for block in (anonymous namespace)::ARKitXRCameraProvider::ScheduleReleaseRetainedTextures() + 383 (ARKitXRCameraProvider.mm:383) 2 Metal 0x22430 MTLDispatchListApply + 52 3 Metal 0xd340 -[_MTLCommandBuffer didCompleteWithStartTime:endTime:error:] + 520 4 IOGPU 0x4b04 -[IOGPUMetalCommandBuffer didCompleteWithStartTime:endTime:error:] + 220 5 Metal 0x224c4 -[_MTLCommandQueue commandBufferDidComplete:startTime:completionTime:error:] + 108 6 IOGPU 0x11c14 IOGPUNotificationQueueDispatchAvailableCompletionNotifications + 128 7 IOGPU 0x594c __IOGPUNotificationQueueSetDispatchQueue_block_invoke + 64 8 libdispatch.dylib 0x409c _dispatch_client_callout4 + 20 9 libdispatch.dylib 0x20830 _dispatch_mach_msg_invoke + 468 10 libdispatch.dylib 0xb56c _dispatch_lane_serial_drain + 376 11 libdispatch.dylib 0x2154c _dispatch_mach_invoke + 448 12 libdispatch.dylib 0xb56c _dispatch_lane_serial_drain + 376 13 libdispatch.dylib 0xc214 _dispatch_lane_invoke + 436 14 libdispatch.dylib 0xb56c _dispatch_lane_serial_drain + 376 15 libdispatch.dylib 0xc1e0 _dispatch_lane_invoke + 384 16 libdispatch.dylib 0x16e10 _dispatch_workloop_worker_thread + 652 17 libsystem_pthread.dylib 0xdf8 _pthread_wqthread + 288 18 libsystem_pthread.dylib 0xb98 start_wqthread + 8
@XubeiX - Thanks for the stack trace. When do you see these crashes, when AR Scene is starting, actively running or exiting?
I have also been testing the fix. I can confirm that the jittery camera feed has been resolved but I am also getting random crashing with the same stack trace. This is while AR is actively running.
Version info: Unity 2021.3.9f1 ARFoundation 4.2.6 Version 14.0 (14A309)
Stack Trace:
thread #59, queue = 'com.Metal.CompletionQueueDispatch', stop reason = EXC_BREAKPOINT (code=1, subcode=0x1a6aacd44) frame #0: 0x00000001a6aacd44 CoreFoundation
CFRelease + 88 frame #1: 0x000000010d0f9ad0 UnityFrameworkinvocation function for block in (anonymous namespace)::ARKitXRCameraProvider::ScheduleReleaseRetainedTextures() + 24 frame #2: 0x00000001a1821430 MetalMTLDispatchListApply + 52 frame #3: 0x00000001a180c340 Metal-[_MTLCommandBuffer didCompleteWithStartTime:endTime:error:] + 520 frame #4: 0x000000020d732b04 IOGPU-[IOGPUMetalCommandBuffer didCompleteWithStartTime:endTime:error:] + 220 frame #5: 0x00000001a18214c4 Metal-[_MTLCommandQueue commandBufferDidComplete:startTime:completionTime:error:] + 108 frame #6: 0x000000020d73fc14 IOGPUIOGPUNotificationQueueDispatchAvailableCompletionNotifications + 128 frame #7: 0x000000020d73394c IOGPU__IOGPUNotificationQueueSetDispatchQueue_block_invoke + 64 frame #8: 0x00000001adeb209c libdispatch.dylib_dispatch_client_callout4 + 20 frame #9: 0x00000001adece830 libdispatch.dylib_dispatch_mach_msg_invoke + 468 frame #10: 0x00000001adeb956c libdispatch.dylib_dispatch_lane_serial_drain + 376 frame #11: 0x00000001adecf54c libdispatch.dylib_dispatch_mach_invoke + 448 frame #12: 0x00000001adeb956c libdispatch.dylib_dispatch_lane_serial_drain + 376 frame #13: 0x00000001adeba214 libdispatch.dylib_dispatch_lane_invoke + 436 frame #14: 0x00000001adeb956c libdispatch.dylib_dispatch_lane_serial_drain + 376 frame #15: 0x00000001adeba1e0 libdispatch.dylib_dispatch_lane_invoke + 384 frame #16: 0x00000001adec4e10 libdispatch.dylib_dispatch_workloop_worker_thread + 652 frame #17: 0x00000001f3354df8 libsystem_pthread.dylib_pthread_wqthread + 288
What are your repro steps? Are you using AR Foundation samples or your own project?
I'm unable to reproduce this crash using SimpleAR scene from AR Foundation Samples:
Unity 2021.3.9f1 AR Foundation 4.2.6 Xcode: 14.0 (14A309) iPhone 12 Pro + iOS 16.0 (20A362)
I tried several runs without running into this crash. I also ran the scene for 10 mins straight to check for longer runtime but couldn't get the crash to reproduce.
@ankur-unity At this moment I notice the app random crashes only when AR Scene is starting or actively running. I didn't notice this happening when leaving the AR scene or closing the application.
I noticed the problem in my own project, today I will further try to find the cause of the problem.
@ankur-unity
After installing the updates to ARKit and AR Foundation, the camera feed now runs at an unusable frame rate (about 1 frame every 3 seconds). Will there be any more updates in the near future or have any other users reported this issue?
Unity 2021.3.6f1 ARKit 4.2.6 AR Foundation 4.2.6 Xcode 13.4.1 iPhone 12 Mini iOS 16.0
I'm running 4.1.12 for both ARKit and ARFoundation and that fixed it for me, everything looks normal so far. I'll be testing a lot tomorrow and I'll report back here.
@ankur-unity
After installing the updates to ARKit and AR Foundation, the camera feed now runs at an unusable frame rate (about 1 frame every 3 seconds). Will there be any more updates in the near future or have any other users reported this issue?
@AlecCDAS No, this is not a known issue. 1 frame / 3 sec sounds really low.
- Is this with the AR Foundation samples or in your own project?
- Is it only reproducible with ARKit 4.2.6 and not with 4.2.3 for the same project, editor, Xcode, and iPhone 12 Mini?
- Do you see any errors in the Xcode log window?
@ankur-unity
After installing the updates to ARKit and AR Foundation, the camera feed now runs at an unusable frame rate (about 1 frame every 3 seconds). Will there be any more updates in the near future or have any other users reported this issue?
Unity 2021.3.6f1 ARKit 4.2.6 AR Foundation 4.2.6 Xcode 13.4.1 iPhone 12 Mini iOS 16.0
I have this same problem, the camera instability is fixed, but now when I restart the scene with an ARSession the frame rate drops very low (unusable)
Unity 2020.3.34f1 ARKit 4.2.6 and 4.1.12 (I tested both of them) AR Foundation 4.2.6 and 4.1.12 (I tested both of them) Xcode 14.0 iPhone 12 Pro Max iOS 16
(Edit)
I have tried with arfoundation-examples the version for Unity 2022 and 2020, and the problem does not happen. It must be something we don't do like you.
@ankur-unity
This is in our own project. If I use the version of my project that is on 4.2.3 (same Xcode and device as well) the only issue is the unstable camera feed due to the iOS 16 issue you had already mentioned above.
We have narrowed it down to the XRCameraSubsystem.TryGetLatestFrame() is running significantly slower than in previous versions. If we skip this method, everything runs as intended. Previously this method did not have any performance issues.
https://docs.unity3d.com/Packages/[email protected]/api/UnityEngine.XR.ARSubsystems.XRCameraSubsystem.html#UnityEngine_XR_ARSubsystems_XRCameraSubsystem_TryGetLatestFrame_UnityEngine_XR_ARSubsystems_XRCameraParams_UnityEngine_XR_ARSubsystems_XRCameraFrame__
@ankur-unity
I have forked the sample repo and added the TryGetLatestFrame() call to the CpuImages scene. The scene was running as intended before adding this method and performance completely tanks after adding it.
https://github.com/AlecCDAS/TryGetLatestFrame_Test
@AlecCDAS - I cannot reproduce the frame rate issue using the AR Foundation Samples with following:
Unity 2021.3.10f1 ARKit 4.2.6 AR Foundation 4.2.6 Xcode 13.4.1 (13F100) iPhone 12 Pro + iOS 16.0
I will try your fork. Thanks for providing that. In the meantime, can you file a bug for it so that there is a public issue tracker to track the progress? https://unity3d.com/unity/qa/bug-reporting
@ankur-unity I have filed a bug on this exact issue through the Unity interface as suggested but never received a confirmation email. I can't give you any identifying info as I have no email.
@ankur-unity
After installing the updates to ARKit and AR Foundation, the camera feed now runs at an unusable frame rate (about 1 frame every 3 seconds). Will there be any more updates in the near future or have any other users reported this issue?
Unity 2021.3.6f1 ARKit 4.2.6 AR Foundation 4.2.6 Xcode 13.4.1 iPhone 12 Mini iOS 16.0
I have this same problem, the camera instability is fixed, but now when I restart the scene with an ARSession the frame rate drops very low (unusable)
Unity 2020.3.34f1
ARKit 4.2.6 and 4.1.12 (I tested both of them)
AR Foundation 4.2.6 and 4.1.12 (I tested both of them)
Xcode 14.0
iPhone 12 Pro Max iOS 16
(Edit)
I have tried with arfoundation-examples the version for Unity 2022 and 2020, and the problem does not happen. It must be something we don't do like you.
After testing for a long time on my project, I have found the cause of the drop in frames. In our application we disable the AR Foundation camera when we are not using it, this causes the frames per second to drop to what is quoted by @AlecCDAS when the AR Foundation camera is enable again.
This has only happened to us with recent AR Foundation versions.
hi, getting the same crashes as mentioned above running Unity 2021.3.9f1 with ARFoundation 4.2.6 on iOS 16 with iPhone X:
Thread 46 Crashed:
0 libobjc.A.dylib 0x00000001afac9c88 objc_msgSend + 136
1 UnityFramework 0x000000010870d0bc invocation function for block in (anonymous namespace)::ARKitXRCameraProvider::ScheduleReleaseRetainedTextures() + 32 (ARKitXRCameraProvider.mm:399)
2 Metal 0x00000001b12f2dac MTLDispatchListApply + 44 (MTLCommandBuffer.m:59)
3 Metal 0x00000001b12de55c -[_MTLCommandBuffer didCompleteWithStartTime:endTime:error:] + 516 (MTLCommandBuffer.m:1053)
4 IOGPU 0x0000000217a691b8 -[IOGPUMetalCommandBuffer didCompleteWithStartTime:endTime:error:] + 216 (IOGPUMetalCommandBuffer.m:157)
5 Metal 0x00000001b12f2e3c -[_MTLCommandQueue commandBufferDidComplete:startTime:completionTime:error:] + 104 (MTLCommandQueue.m:597)
6 IOGPU 0x0000000217a7592c IOGPUNotificationQueueDispatchAvailableCompletionNotifications + 120 (IOGPUNotificationQueue.c:331)
7 IOGPU 0x0000000217a69f64 __IOGPUNotificationQueueSetDispatchQueue_block_invoke + 60 (IOGPUNotificationQueue.c:251)
8 libdispatch.dylib 0x00000001bd1ea87c _dispatch_client_callout4 + 16 (object.m:600)
9 libdispatch.dylib 0x00000001bd1bd7c0 _dispatch_mach_msg_invoke$VARIANT$armv81 + 372 (mach.c:2462)
10 libdispatch.dylib 0x00000001bd1c572c _dispatch_lane_serial_drain$VARIANT$armv81 + 308 (inline_internal.h:0)
11 libdispatch.dylib 0x00000001bd1be35c _dispatch_mach_invoke$VARIANT$armv81 + 444 (mach.c:2784)
12 libdispatch.dylib 0x00000001bd1c572c _dispatch_lane_serial_drain$VARIANT$armv81 + 308 (inline_internal.h:0)
13 libdispatch.dylib 0x00000001bd1c6318 _dispatch_lane_invoke$VARIANT$armv81 + 432 (queue.c:3940)
14 libdispatch.dylib 0x00000001bd1c572c _dispatch_lane_serial_drain$VARIANT$armv81 + 308 (inline_internal.h:0)
15 libdispatch.dylib 0x00000001bd1c62e4 _dispatch_lane_invoke$VARIANT$armv81 + 380 (queue.c:3940)
16 libdispatch.dylib 0x00000001bd1d0000 _dispatch_workloop_worker_thread + 612 (queue.c:6846)
17 libsystem_pthread.dylib 0x00000001fd700b50 _pthread_wqthread + 284 (pthread.c:2618)
18 libsystem_pthread.dylib 0x00000001fd70067c start_wqthread + 8
any help with this would be appreciated.
I had many crash problems in the App, until I deactivated Multithreading in Player Settings:
After deactivating it, the crashes of the App were considerably reduced. I runing in iPhone 12 Pro Max
I had many crash problems in the App, until I deactivated Multithreading in Player Settings:
After deactivating it, the crashes of the App were considerably reduced. I runing in iPhone 12 Pro Max
thanks, but won’t this have a heavy impact on performance? from an official unity tutorial:
You should enable Multithreaded Rendering whenever possible, as it usually benefits performance greatly.
and before updating to iOS 16 I almost never had app crashes.
After several days of testing, I can also confirm - disabling multi-threaded rendering significantly reduces the problem. In my case, the problem stopped occurring - Crashlytics has not currently recorded this error once.
Previously in the project I always had multithreaded rendering enabled and there were no problems.
thanks @PabloBuitrago and @XubeiX for your advice. I now also disabled multi-threaded rendering and so far no more crash happened. Also, I didn’t notice any impact on performance. So going with this until the underlying problem is fixed
Hi
Tried to update to the following versions of ARFoundation and ARKit: 4.2.6 4.1.12
They both have low FPS when using it. Only ARFoundation and ARKit version 4.2.3 are providing reaching FPS target.
Config: Unity 2021.3.10 XCode 14 IPhone SE2 with iOS 16
Hi
Tried to update to the following versions of ARFoundation and ARKit:
4.2.6
4.1.12
They both have low FPS when using it. Only ARFoundation and ARKit version 4.2.3 are providing reaching FPS target.
Config:
Unity 2021.3.10
XCode 14
IPhone SE2 with iOS 16
In your project do you disable the AR Foundation camera at some point?
Hi Tried to update to the following versions of ARFoundation and ARKit: 4.2.6 4.1.12 They both have low FPS when using it. Only ARFoundation and ARKit version 4.2.3 are providing reaching FPS target. Config: Unity 2021.3.10 XCode 14 IPhone SE2 with iOS 16
In your project do you disable the AR Foundation camera at some point?
Hi No I don't, I made sure of that Unfortunately I cannot update to 5.0.2 because of a compatibility with another package using ARFoundation
I notice the same crashes at ARKitXRCameraProvider::ScheduleReleaseRetainedTextures, especially when using the Face Camera.
Has anybody have the issue that with ARfoundation 4.2.6 and Xcode 14.0 to iOS 16.0.2 the project is no longer able to build since there are undefined symbols?
error build: Undefined symbol: _UnityARKit_Camera_GetTextureReleaseCallbackHandle
Has anybody have the issue that with ARfoundation 4.2.6 and Xcode 14.0 to iOS 16.0.2 the project is no longer able to build since there are undefined symbols?
error build: Undefined symbol: _UnityARKit_Camera_GetTextureReleaseCallbackHandle
@d4n3x Have you updated ARKit XR Plugin to 4.2.6 as well?
Hi
Quick update Shaky camera issue is fixed for me with ARFoundation 4.2.3 and iOS 16.0.2 Apple released an update