Augmented Face detection lips sensitivity

[onuralpszr]

SPECIFIC ISSUE ENCOUNTERED

Hello, I was developing an app based on kotlin/Arcore related around Augmented Face. While I was doing detection of faces I encountered a problem makes a problem while I was doing occlusion around mouth side of the face. Specifically on upper lips section.

Working example based on mediapipe : https://storage.googleapis.com/tfjs-models/demos/face-landmarks-detection/index.html?model=mediapipe_face_mesh

Also the feature it uses for making right : https://google.github.io/mediapipe/solutions/face_mesh.html#refine_landmarks

When Aug. Face detection done and start tracking, I can clearly see that my upper half of the lips on face mesh goes up. and It also effect that any object I put around mouth basically show more mesh then I didn't wanted to see it.

Arcore detection and drawing 468 nodes

Mediapipe detection - Face mesh example without change

VERSIONS USED

• Android Studio: Android Studio Chipmunk | 2021.2.1 Build #AI-212.5712.43.2112.8512546, built on April 28, 2022 Runtime version: 11.0.12+0-b1504.28-7817840 amd64 VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.

• ARCore SDK for Android: ARCore 1.31.0

• Device manufacturer, model, and O/S: Samsung S21 Ultra 5G

• Google Play Services for AR (ARCore): ARCore 1.32.0 service

Packages: versionName=1.32.221520093 Hidden system packages: versionName=0 # of packages: 1 / # of packages in DATA: 1 (4) Active APEX packages: Inactive APEX packages: Factory APEX packages:

• Output of adb shell getprop ro.build.fingerprint: samsung/p3sxtur/p3s:12/SP1A.210812.016/G998BXXU5CVF2:user/release-keys

Snippet code I use for creating those nodes on Augmented Face

             private var sphereNodes: ArrayList<Node> = ArrayList()

            for (i in 0..467) {
                val sphereNode = Node()
                val colorSelect = Color(android.graphics.Color.MAGENTA)
                MaterialFactory.makeOpaqueWithColor(
                    context,
                    colorSelect
                )
                    .thenAccept { material: Material? ->
                        sphereNode.parent = this
                        sphereNode.renderable = ShapeFactory.makeSphere(
                            0.0011f, Vector3(
                                0f,
                                0f,
                                0f
                            ), material
                        )
                        sphereNodes.add(sphereNode)
                    }
            }
            
            
            
       

// For tracking locations ;
// override fun onUpdate(frameTime: FrameTime?) 

            val buffer = augmentedFace!!.meshVertices
            if (buffer != null) {
                sphereNodes.forEachIndexed { index, element ->
                    element.localPosition = Vector3(
                        buffer.get(index * 3),
                        buffer.get(index * 3 + 1),
                        buffer.get(index * 3 + 2)
                    )
                }
            }

I also placed object for testing to see "occlusion" in face, indeed ıt does follow exact same spheres for hiding object or showing object

WORKAROUNDS (IF ANY)

• I didn't find any yet Arcore but I saw that medpipe has refine_landmarks feature for handle that problem and put upper lips below limit exactly stay on upper-lips down side of the lips and create proper border

No refine mode

With refine mode

Source link : https://storage.googleapis.com/tfjs-models/demos/face-landmarks-detection/index.html?model=mediapipe_face_mesh

Maybe there is a way to handle that detection for like "refine-Landmarks" but I didn't find anything yet. Any help would be awesome. Also please let me know If you need anything further

Thank you.

[onuralpszr]

Extra note : I was able to reproduce that problem in "augmented Face example by using" face texture map and I can clearly see that when I smile, "texture moves on top of half my upper lips and think that lower half is "inside of the mouth" as well.

[onuralpszr]

@devbridie any idea ? (friendly ping :) )

[kyhoolee]

I face the same issue too, I think the dev team does not spend much effort for this project. It's really really difficult to combine preview feature and record feature too.

[onuralpszr]

I face the same issue too, I think the dev team does not spend much effort for this project. It's really really difficult to combine preview feature and record feature too.

I found my workaround (not arcore solution) with mediapipe but I customized (face mesh and attention landmark enabled, but issue over there was jitter and I am doing adding filter(like one euro filter, default It was using vector filter for smooting out) and adding extra logic prevent to do that, so It will be act more smoothly but still arcore need something similar with low jitter and lag because this is really problematic

[kyhoolee]

• I am following this raw solution - mediapipe unity plugin https://github.com/keijiro/FaceLandmarkBarracuda
• I can replicate his result with face_landmark.tflite - but error with face_landmark_with_attention when converting to onnx format

!tflite2onnx face_landmark_with_attention.tflite face_landmark.onnx
Traceback (most recent call last):
  File "/usr/local/bin/tflite2onnx", line 8, in <module>
    sys.exit(cmd_convert())
  File "/usr/local/lib/python3.7/dist-packages/tflite2onnx/convert.py", line 58, in cmd_convert
    convert(args.tflite_path, args.onnx_path)
  File "/usr/local/lib/python3.7/dist-packages/tflite2onnx/convert.py", line 44, in convert
    model.convert(explicit_layouts)
  File "/usr/local/lib/python3.7/dist-packages/tflite2onnx/model.py", line 39, in convert
    self.parse()
  File "/usr/local/lib/python3.7/dist-packages/tflite2onnx/model.py", line 31, in parse
    g.parse()
  File "/usr/local/lib/python3.7/dist-packages/tflite2onnx/graph.py", line 63, in parse
    op = self.OPCFactory.create(i)
  File "/usr/local/lib/python3.7/dist-packages/tflite2onnx/op/common.py", line 154, in create
    raise NotImplementedError("Unsupported TFLite OP: {} {}!".format(opcode, name))
NotImplementedError: Unsupported TFLite OP: 32 CUSTOM!