Augmented Face - Not Rendering Model and Still in "surface mode"

[onuralpszr]

Hello, I was trying to understand learn and understand "sceneview" usages, I already discovered how to get Augmented face in "sceneform-android" but when I try sceneview, I encountered several problems,

• First camera settings are seems to be not working properly because some options act wrong. (For example MESH3D mode or I was able to enable Deph mode (normally I shouldn't be able to do that in Front Camera) )
• Model size is too small
• Model moving opposite side when I move my head (If I go right, model goes left)
• Model not render properly and shows me "black material"
• I'm always seeing "Searching for surfaces..." (front camera or back camera) It should not be visible in front camera

Here my minimal code (only MainActivity)

    private lateinit var binding: ActivityMainBinding
    private val facesNodes = HashMap<AugmentedFace, AugmentedFaceNode>()
    private var faceModel: ModelRenderable? = null
    
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        binding = ActivityMainBinding.inflate(layoutInflater)
        setContentView(binding.root)
        loadModels()

    }
    
    private fun loadModels() {

        lifecycleScope.launchWhenCreated {
            binding.sceneView.configureSession { arSession, config ->
                val filter =
                    CameraConfigFilter(arSession).setFacingDirection(CameraConfig.FacingDirection.FRONT)
                val cameraConfig = arSession.getSupportedCameraConfigs(filter)[0]
                arSession.cameraConfig = cameraConfig
                config.instantPlacementMode = Config.InstantPlacementMode.DISABLED

                config.augmentedFaceMode = Config.AugmentedFaceMode.MESH3D
                arSession.configure(config)
            }


            faceModel = loadModel(
                context = this@MainActivity,
                lifecycle = lifecycle,
                glbFileLocation = "models/fox.glb",
            )

            binding.sceneView.onAugmentedFaceUpdate = {
                val existingFaceNode = facesNodes[it]
                when (it.trackingState) {
                    TrackingState.TRACKING ->
                        if (existingFaceNode == null) {
                            val faceNode = AugmentedFaceNode(binding.sceneView.lifecycle, it)
                            val modelInstance = faceNode.setFaceRegionsRenderable(faceModel)
                            modelInstance.isShadowCaster = false
                            modelInstance.isShadowReceiver = true
                            faceNode.faceRegionsRenderable = faceModel!!
                            binding.sceneView.addChild(faceNode)
                            facesNodes[it] = faceNode
                        }
                    TrackingState.STOPPED -> {
                        if (existingFaceNode != null) {
                            binding.sceneView.removeChild(existingFaceNode)
                            facesNodes.remove(it)
                        }
                        facesNodes.remove(it)
                    }
                    else -> {
                        facesNodes.remove(it)
                    }
                }
            }

        }
    }

activity_main.xml snippet as well

    <io.github.sceneview.ar.ArSceneView
        android:id="@+id/sceneView"
        android:layout_width="0dp"
        android:layout_height="0dp"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintLeft_toLeftOf="parent"
        app:layout_constraintRight_toRightOf="parent"
        app:layout_constraintTop_toTopOf="parent" />

PS : I used "fox and "face" glb files from sceneform for quick testing.

If you need additional information please let me know.

Thank you.

[ThomasGorisse]

Hi,

You are doing it the right way but the fact is that we didn't have time yet to work on the AugmentedFaceNode.

Would you mind/like creating a PR on moving the AugmentedFaceNode to Kotlin? That shouldn't be difficult and we will help you on it.

Thanks

[onuralpszr]

Hi @ThomasGorisse

I would like to help about augmentedFaceNode, but first I need better understanding about AugmentedFaceNode, and I'm definitely need help about it.

[onuralpszr]

I was working on AugmentedFaceNode and when I upgrade my version into 0.6.0 now I'm getting texture error as well. Any thoughts ? I can even reproduce same code above and all I did was upgrade version 0.6.0 and texture crashes started.

2022-06-02 19:56:42.081 31880-31880/io.github.sceneview.sample.augmentedfaces E/native: E0000 00:00:1654189002.081197   31880 error_policy_util.cc:263] 
    ################ ARCore Native Error ##################
    BUILD_CHANGELIST:445555480
    BUILD_BASELINE_CHANGELIST:441227770
    ################### Stack Trace Begin ################
    ARCoreError: third_party/arcore/ar/core/frame_manager.cc:57	https://cs.corp.google.com/piper///depot/google3/third_party/arcore/ar/core/frame_manager.cc?g=0&l=57
    ARCoreError: third_party/arcore/ar/core/session.cc:1919	https://cs.corp.google.com/piper///depot/google3/third_party/arcore/ar/core/session.cc?g=0&l=1919
    ARCoreError: third_party/arcore/ar/core/c_api/session_lite_c_api.cc:76	https://cs.corp.google.com/piper///depot/google3/third_party/arcore/ar/core/c_api/session_lite_c_api.cc?g=0&l=76
    ################### Stack Trace End #################
    
    ################### Undecorated Trace Begin  #################
    FAILED_PRECONDITION: 
    ARCoreError: third_party/arcore/ar/core/session.cc:1919
    
    ARCoreError: third_party/arcore/ar/core/frame_manager.cc:57
    texture names are not set. [type.googleapis.com/util.ErrorSpacePayload='ArStatusErrorSpace::AR_ERROR_TEXTURE_NOT_SET']
    === Source Location Trace: === 
    third_party/arcore/ar/core/status.cc:171
    third_party/arcore/ar/core/frame_manager.cc:57
    third_party/arcore/ar/core/session.cc:1919
    
    ################### Undecorated Trace End  #################
2022-06-02 19:56:42.081 31880-31880/io.github.sceneview.sample.augmentedfaces D/AndroidRuntime: Shutting down VM
2022-06-02 19:56:42.082 31880-31880/io.github.sceneview.sample.augmentedfaces E/AndroidRuntime: FATAL EXCEPTION: main
    Process: io.github.sceneview.sample.augmentedfaces, PID: 31880
    com.google.ar.core.exceptions.TextureNotSetException
        at java.lang.reflect.Constructor.newInstance0(Native Method)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:343)
        at com.google.ar.core.Session.throwExceptionFromArStatus(Session.java:14)
        at com.google.ar.core.Session.nativeUpdate(Native Method)
        at com.google.ar.core.Session.update(Session.java:2)
        at io.github.sceneview.ar.arcore.ArSession.update(ArSession.kt:106)
        at io.github.sceneview.ar.ArSceneView.doFrame(ArSceneView.kt:359)
        at io.github.sceneview.SceneView.doFrame(SceneView.kt:330)
        at android.view.Choreographer$CallbackRecord.run(Choreographer.java:1106)
        at android.view.Choreographer.doCallbacks(Choreographer.java:866)
        at android.view.Choreographer.doFrame(Choreographer.java:792)
        at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:1092)
        at android.os.Handler.handleCallback(Handler.java:938)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loopOnce(Looper.java:226)
        at android.os.Looper.loop(Looper.java:313)
        at android.app.ActivityThread.main(ActivityThread.java:8669)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:571)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1135)

[onuralpszr]

I can trigger this version 0.5.2 and latest but 0.5.1 is working and NOT giving me Texture error.

[tribetomi]

@onuralpszr

Have you had any luck in fixing this issue?

[onuralpszr]

@onuralpszr

Have you had any luck in fixing this issue?

I made bit more progress on this and trying to stable somehow so I can share with PR and repo so everybody try it (of course try with care, it can be break :)) )

[ThomasGorisse]

Hi @onuralpszr and @tribetomi

Sorry for the very late answer due to focusing on other main parts of SceneView.

@onuralpszr You well analysed current issues from the Augmented Face usage with last SceneView. They are mostly here because of not integrated ArFrontFacingFragment within SceneView.

If you are both ready to help, I gave you access to SceneView Extended. This repo is private and contains specific samples like AugmentedImageNode, soon ML sample and that should also contain AugmentedFaceNode. Access to it is limited to code or financial donators/contributors for now and we will maybe make it public within SceneView a little later.

So, if you want to contribute, here are the tasks:

• Copy/Paste Augmented Face from Sceneform to sceneviewextended/AugmentedFaceNode
• Convert to Kotlin SceneView Node (you can look at AugmentedImageNode and do the same)
• Move from Sceneform calculation to Kotlin-math (don't be afraid about this part, just copy/paste those code parts within Discord, @grassydragon and I will give you the Kotlin-math translate)
• Move missing code parts from ArFrontFacingFragment(Config for ARCore and Filament front camera usage)
• Move Sceneform Augmented Face sample to SceneView Extended

Let me know if you want to do it and don't be afraid of seems complicated parts we are here to answer as quick as possible.

Thanks

[tribetomi]

Hey @ThomasGorisse , thanks for answering. I am happy to do it. Will ping you if I get stuck!

[tribetomi]

@ThomasGorisse

I added new branch with AugmentedFaceNode -> new branch , you may check it out

I hope that's alright. It's still wip obviously because I am still in the process of understanding new SceneView in general.

Some of the issue here:

• usage of Ar, Model or basic Node? Evene regular node doesn't have renderable setter?
• do we need augmentedImageDatabase for saving frames?
• are we using old Engine(Instance) for transformation? And Matrix from Sceneform.math since I haven't found one from Romain Guy?

Just few questions for start

[onuralpszr]

@ThomasGorisse

I added new branch with AugmentedFaceNode -> new branch , you may check it out

I hope that's alright. It's still wip obviously because I am still in the process of understanding new SceneView in general.

Some of the issue here:

• usage of Ar, Model or basic Node? Evene regular node doesn't have renderable setter?
• do we need augmentedImageDatabase for saving frames?
• are we using old Engine(Instance) for transformation? And Matrix from Sceneform.math since I haven't found one from Romain Guy?

Just few questions for start

• That's because (afaik) you need to add those in "sceneview-extended" because in node folder all I saw was video and image nodes rest of them wasn't exist yet.
• you don't need AugImageDatabase
• https://github.com/romainguy/kotlin-math
• For the Augmented Face stuff you gonna need to handle "occlude" material side as well (because pure augmented face from arcore doesn't hide anything behind your face or head

[tribetomi]

For occluder I think I can use occluder mesh. Something like:

     Material.builder()
        .setSource(buffer)
        .build(lifecycle)
        .thenAccept { mat ->
          occluderMask = mat
        }

[ThomasGorisse]

Have a look at the MaterialLoader class instead

[ThomasGorisse]

I had a first look at your code and it's looking in a good way. The main goal would be not to have any imports from the old Sceneform package but I also know that it needs a little work to find to the actual Filament calls coming from the Augmented face RenderableDefinition calls.

Those Filament Mesh Builder direct calls must be included inside the AugmentedFaceNode

[tribetomi]

@onuralpszr do you have any update perhaps?

I just pushed update where AugmentedFaceNode is copied without errors and warning. I added simple session code to MainFragment to try it but I keep getting:

com.google.ar.core.exceptions.TextureNotSetException

In the old SceneView there was a flag

// Session needs access to a texture id for updating the camera stream. // Filament and the Main thread each have their own gl context that share resources for this. // Reset the hasSetTextureNames variable so that the texture name is set during the first call to onBeginFrame. hasSetTextureNames = false;

I can't find that one in new ArSceneView.

@ThomasGorisse what should be done with texture?

[ThomasGorisse]

Hi @tribetomi. First, can you try moving to ArSceneView 0.8.0? It introduces the RIP Renderer part so it's clearer to see where the texture issue is happening. The new Filament Texture import mechanism added changes around the gl texture but I don't think it's related. Do you modify the session config before or after onResume?

[ThomasGorisse]

Having a first look from my phone, you may remove the launchOnCreated around the session configuration.

[tribetomi]

@ThomasGorisse I updated lib and moved session creation to onResume(before it was in onViewCreated so before) but crash remains:

com.google.ar.core.exceptions.TextureNotSetException at java.lang.reflect.Constructor.newInstance0(Native Method) at java.lang.reflect.Constructor.newInstance(Constructor.java:343) at com.google.ar.core.Session.throwExceptionFromArStatus(Session.java:14) at com.google.ar.core.Session.nativeUpdate(Native Method) at com.google.ar.core.Session.update(Session.java:2) at io.github.sceneview.ar.arcore.ArSession.update(ArSession.kt:106) at io.github.sceneview.ar.ArSceneView.doFrame(ArSceneView.kt:378) at io.github.sceneview.SceneView.doFrame(SceneView.kt:480) at android.view.Choreographer$CallbackRecord.run(Choreographer.java:1106) at android.view.Choreographer.doCallbacks(Choreographer.java:866) at android.view.Choreographer.doFrame(Choreographer.java:792) at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:1092) at android.os.Handler.handleCallback(Handler.java:938) at android.os.Handler.dispatchMessage(Handler.java:99) at android.os.Looper.loopOnce(Looper.java:226) at android.os.Looper.loop(Looper.java:313) at android.app.ActivityThread.main(ActivityThread.java:8663) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:567) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1135)

Debugging it I don't see at which point this texture is failing and where exactly it is being set.

EDIT: I saw that config must be called before onResume so I moved it back

[tribetomi]

@ThomasGorisse @grassydragon After a bit of investigation I think there is not much that can be done from outside of lib.

In ArCameraStream on line 167 GLHelper calls createCameraTexture. It happens right on first init of ArCameraStream that is init in ArSceneView(215). It is init with default camera that is 0/BACK. Once you change cameraConfig those resources are lost and this error appears.

It is the same issue as in: https://github.com/google-ar/arcore-android-sdk/issues/1170#issuecomment-825339396

Only thing from outside that can be done is to destroy the stream. hasSetTextureNames is private, createCameraTexture has default cameraId, etc.

[tribetomi]

@ThomasGorisse can I expect some update regarding this matter?

[onuralpszr]

For my end, I had a busy month and it is getting close to finish. I will come up with update and code change, for texture problem I had to check so I can't answer that one atm

[grassydragon]

It is the same issue as in: https://github.com/google-ar/arcore-android-sdk/issues/1170#issuecomment-825339396

I think that explanation there is a little bit wrong. The texture ID points to the OpenGL texture but it is up to ARCore to actually create this texture and fill it with camera frames. When setTextureName is called we connect the texture with the camera that is currently specified by the ARCore CameraConfig. Therefore, the texture and camera IDs don't have to match. Moreover, when using multiple textures with setTextureNames each texture from the array has its own ID.

It can be so that calling setCameraConfig resets the texture names and we need to set them again, for example, by setting hasSetTextureNames to false.

[github-actions[bot]]

Hey there, it looks like there has been no activity on this issue recently. Has the issue been fixed, or does it still require the community's attention? This issue may be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

Closing this issue after a prolonged period of inactivity. If this issue is still present in the latest release, please feel free to create a new issue with up-to-date information.

[eksizadam]

Hello I find the solution, you need to put this after config change

sceneView.onArSessionResumed = { it.setCameraTextureNames(sceneView.arCameraStream.cameraTextureIds) }