AR Camera FPS drop after return back to the screen with UnityView

[serdarcevher]

This is only happening in Android - iOS is fine.

I open the StreetScreen having the UnityView component. Running okay. Go back home, then navigate to StreetScreen again. AR Camera FPS is significantly reducing.

(I use androidKeepPlayerMounted={true}, otherwise the UnityView comes blank after refocusing on the screen.)

• Tried to set the screen to be unmounted on blur, didn't have any effect.
• Tried unloading Unity manually, it remained blank.
• Tried to run pauseUnity and resumeUnity when blurred/focused, didn't have any effect.

Does anyone have any idea about how to find a solution to this? (screen video attached - FPS drop starts at 24th second)

https://github.com/azesmway/react-native-unity/assets/824239/525ddb51-bdb2-41d0-8c98-9287c0b42293

[Luluno01]

Had exactly the same problem (and the black screen problem). I solved this by fully reloading the scene and reinitializing AR Foundation after reattaching UnityView. More precisely, I deinitialize AR Foundation before detaching UnityView, and reinitialize UnityView after the auto scene reloading after reattaching UnityView.

Here are few things that I found (not sure if they still apply):

• Unity seems to automatically reload the scene after reattaching.
• AR Foundation will not auto reinitialize between scene reloads.
• When you put AR Foundation running in the background, because Unity pauses itself while in the background, the video stream input to AR foundation will be paused, too. This confuses AR Foundation and ultimately leads to broken tracking (unable to infer where to put the ARPlanes, etc.) and FPS drop when you get back.
• You need to manually sync the messaging between RN and Unity. The order of onUnityMessage callback being available and the scene finishing loading is not guaranteed. My approach is to implement a handshake mechanism to make sure both RN and Unity are fully loaded before doing any other work.

[serdarcevher]

Had exactly the same problem (and the black screen problem). I solved this by fully reloading the scene and reinitializing AR Foundation after reattaching UnityView. More precisely, I deinitialize AR Foundation before detaching UnityView, and reinitialize UnityView after the auto scene reloading after reattaching UnityView.

Here are few things that I found (not sure if they still apply):

• Unity seems to automatically reload the scene after reattaching.
• AR Foundation will not auto reinitialize between scene reloads.
• When you put AR Foundation running in the background, because Unity pauses itself while in the background, the video stream input to AR foundation will be paused, too. This confuses AR Foundation and ultimately leads to broken tracking (unable to infer where to put the ARPlanes, etc.) and FPS drop when you get back.
• You need to manually sync the messaging between RN and Unity. The order of onUnityMessage callback being available and the scene finishing loading is not guaranteed. My approach is to implement a handshake mechanism to make sure both RN and Unity are fully loaded before doing any other work.

Thank you very much for your instructions. As you pointed out, I'll try implementing an handshake mechanism when I return back to work.

[serdarcevher]

Hey @Luluno01, how do you deinitialize AR Foundation before detaching UnityView?

I try running Application.Quit() in Unity, but this closes the React Native app too.

[Luluno01]

Hey @Luluno01, how do you deinitialize AR Foundation before detaching UnityView?

I try running Application.Quit() in Unity, but this closes the React Native app too.

Something like this: https://docs.unity3d.com/Packages/[email protected]/api/UnityEngine.XR.ARFoundation.LoaderUtility.Deinitialize.html

---

Found the precise code I use:

using System;
using UnityEngine;
using UnityEngine.XR.ARFoundation;
using Cysharp.Threading.Tasks;

public class SomeLifecycleManager : MonoBehaviour
{
    // ...
    [SerializeField] private ARPlaneManager arPlaneManager;
    [SerializeField] private ARSession arSession;

    public async UniTask UnloadAll()
    {
        // Unload everything necessary to prepare for an automatic scene reload later
        // ...
        // Reset AR session
        arPlaneManager.enabled = false; // Disable the plane manager so that it does not create new ones
        await UniTask.WaitForEndOfFrame(this);
        arSession.Reset();
        // https://forum.unity.com/threads/arfoundation-how-to-properly-remove-planes-and-ref-points.661762/
        // "Call Reset() on your AR Session. Wait a couple of frames. Enable plane manager."
        await UniTask.DelayFrame(15);
        // https://docs.unity3d.com/Packages/[email protected]/api/UnityEngine.XR.ARFoundation.LoaderUtility.html
        LoaderUtility.Deinitialize();
    }
}

[serdarcevher]

Hey @Luluno01, how do you deinitialize AR Foundation before detaching UnityView? I try running Application.Quit() in Unity, but this closes the React Native app too.

Something like this: https://docs.unity3d.com/Packages/[email protected]/api/UnityEngine.XR.ARFoundation.LoaderUtility.Deinitialize.html

Found the precise code I use:

using System;
using UnityEngine;
using UnityEngine.XR.ARFoundation;
using Cysharp.Threading.Tasks;

public class SomeLifecycleManager : MonoBehaviour
{
    // ...
    [SerializeField] private ARPlaneManager arPlaneManager;
    [SerializeField] private ARSession arSession;

    public async UniTask UnloadAll()
    {
        // Unload everything necessary to prepare for an automatic scene reload later
        // ...
        // Reset AR session
        arPlaneManager.enabled = false; // Disable the plane manager so that it does not create new ones
        await UniTask.WaitForEndOfFrame(this);
        arSession.Reset();
        // https://forum.unity.com/threads/arfoundation-how-to-properly-remove-planes-and-ref-points.661762/
        // "Call Reset() on your AR Session. Wait a couple of frames. Enable plane manager."
        await UniTask.DelayFrame(15);
        // https://docs.unity3d.com/Packages/[email protected]/api/UnityEngine.XR.ARFoundation.LoaderUtility.html
        LoaderUtility.Deinitialize();
    }
}

Thank you very much for sharing the precise code.

[serdarcevher]

I ended up by reloading the whole scene + restarting the AR session + passing the data back - in the user leaves the screen and comes back.

So it works as:

const { unityUnloaded, setUnityUnloaded } = useContext(MyContext);

const leaveScreen = () => { if (Platform.OS != 'ios') { console.log('setUnityUnloaded(true) in Android'); setUnityUnloaded(true); } }

I send a message from Unity to React Native after the Awake function runs: "awakeCompleted". I capture it in the React Native side:

if (data.messageType && data.messageType == 'awakeCompleted') { if (Platform.OS == 'android') { if (unityUnloaded) { setUnityUnloaded(false); setTimeout(() => { runInUnity('ReloadAll'); }, 500) } } }

My ReloadAll function in Unity: public void ReloadAll() { SceneManager.UnloadScene("DefaultScene"); SceneManager.LoadScene("DefaultScene"); ARLocationManager.Instance.ResetARSession((() => { Debug.Log("AR+GPS and AR Session were restarted!"); DataToReact.Instance.SendReloadMessage(); })); }

Finally, when the reload message is received from React Native, I pass back the existing data from React Native to the Unity scene:

if (data.messageType && data.messageType == 'reloadCompleted') { console.log('reloadCompleted message received in React Native from Unity'); replaceItemsForAndroid(); //sending JSON data to be placed in the Unity scene }

[Luluno01]

I ended up by reloading the whole scene + restarting the AR session + passing the data back - in the user leaves the screen and comes back.

So it works as: const { unityUnloaded, setUnityUnloaded } = useContext(MyContext); const leaveScreen = () => { ... if (Platform.OS != 'ios') { console.log('setUnityUnloaded(true) in Android'); setUnityUnloaded(true); } }

I send a message from Unity to React Native after the Awake function runs: "awakeCompleted". I capture it in the React Native side:

if (data.messageType && data.messageType == 'awakeCompleted') { if (Platform.OS == 'android') { if (unityUnloaded) { setUnityUnloaded(false); setTimeout(() => { runInUnity('ReloadAll'); }, 500) } } }

My ReloadAll function in Unity: ` public void ReloadAll() { SceneManager.UnloadScene("DefaultScene"); SceneManager.LoadScene("DefaultScene");

ARLocationManager.Instance.ResetARSession((() =>
{
	Debug.Log("AR+GPS and AR Session were restarted!");
	DataToReact.Instance.SendReloadMessage();
}));

} `

Finally, when the reload message is received from React Native, I pass back the existing data from React Native to the Unity scene:

if (data.messageType && data.messageType == 'reloadCompleted') { console.log('reloadCompleted message received in React Native from Unity'); replaceItemsForAndroid(); //sending JSON data to be placed in the Unity scene }

Yes, that's exactly what I suggest. And if you find UnityView still shows all black and is not ticking after being reattached on Android, you may try windowFocusChanged(hasFocus: boolean = false).

---

Here is a short summary of the flow of remounting UnityView with AR Foundation running for those who might have the same problem:

Unloading:

1. RN: send a message to Unity via postMessage to initiate a "stop".
2. Unity: receive the "stop" message, start unloading everything, including AR Foundation.
3. Unity: send a "stopped" message back to RN to tell RN it is safe to unmount.
4. RN: receive the "stopped" message, unmount UnityView

Reloading:

1. RN: mount UnityView, optionally call windowFocusChanged on Android, send a handshake message to Unity via postMessage every few seconds (because UnityView might miss initial messages from RN).
2. Unity: receive the handshake message (and ignore further handshake messages), reload the scene (just for safety, you can also rely on the automatic scene reloading).
3. Unity: upon scene reloaded, send a handshake message to RN to tell RN everything is ready for action.
4. RN: do whatever you want to as both RN and Unity are ready.

You may also implement alternative handshake mechanisms depending on what you need.

[Thomas-akkari]

Hi, I'm facing the same issue with Android, followed the steps above but I'm getting occasional crashes/inconsistent results when mounting back my Unity. I'm guessing it has to do with my unload but can't figure it out. Is there a way to restart the Unity part everytime on mount to have a fresh start somehow ?

[Luluno01]

Hi, I'm facing the same issue with Android, followed the steps above but I'm getting occasional crashes/inconsistent results when mounting back my Unity. I'm guessing it has to do with my unload but can't figure it out. Is there a way to restart the Unity part everytime on mount to have a fresh start somehow ?

No. Unity is not designed to restart without killing the process. I wanted to do exactly what you want to do for pretty much the same reason as yours, but it turns out you simply can't (unless Unity is to implement the support). In the end I found the embedded Unity crashes randomly because of a double-free bug. By solving that bug it no longer crashes.

So if you can find and the cause to the crashes, find and eliminate it (it will crash the game even when running standalone). If it is out of your control, consider restart the entire app instead of the Unity part. If restarting the entire aop is unacceptable, try run UnityActivity in a separate process (see android:process. Note that if you start UnityActivity in a separate process, you will need to implement your own inter-process communication mechanism.

[jb-san]

They way that I solved this was to add

 private void OnApplicationPause(bool pause)
    {

        if (pause)
        {
            try
            {
                arSession.Reset();
                LoaderUtility.Deinitialize();
                LoaderUtility.Initialize();
                SceneManager.LoadScene(SceneManager.GetActiveScene().name);
                
            }
            catch(Exception e)
            {
                _messageHandler.SendConsoleLogToRN($"UNITY: OnApplicationPause {e}");
            }
        }
    }

in a mono behaviour