Invalid mask position and size for portrait mode.

[CheshireCat92]

Hello everybody! I tried to use ARCore Face Recognition as ARKit Face Session replacement in my project. After an example app investigation, I saw that example used CALayer baked to UIView as a video visualizer. I modified the example's code inserted imageCaptureLayer at SCNScene background's content. Also, I modified capture's output settings like added video orientation and disabled video mirroring. After all of these operations mask (default fox mask from example) somewhy rendered always at the left edge of the screen like screen orientation is landscape, not portrait. I fixed it changing orientation to .landscapeRight when using projectionMatrix method. Of course, recognition angles has been changed to correct angles. After all of this, the fox mask rendered so big. It looks like my face's geometry 3x bigger that it's real. Please, help me to understand how can I fix it?

All sources attached bellow. ARCore.zip

[rsfuller]

It looks like you changed the layout to use Interface Builder, but you only included the swift files. Can you upload the whole project? Thanks!

[CheshireCat92]

@rsfuller sure. link to a project on GitHub -> https://github.com/CheshireCat92/ARCore-iOS-

[CheshireCat92]

Hi @rsfuller! Yesterday I updated the source code project (an app didn't start correctly on XCode 11.3), so please, get latest changes from Github's repo.

[coolpolus]

Hello,

Thank you for labeling! Really important moment for us to continue ARCore integration in a project, hope that soon it can be resolved.

Best regards, Ruslan.

[rsfuller]

Is there a specific reason you do not want use the UIView's layer to display the camera feed / video? We set it up this way because it was more performant than any other method from our testing.

If you must display the video inside the scene, I would recommend the following:

• Leave the camera capture video orientation in landscape
• Create a SCNNode with SCNPlane geometry and add it to the scene
• Set the plane's width/height required to display the video fullscreen based on the camera's FOV and the plane's z position [Example]
• Convert the video frame (CVPixelBuffer) to a MTLTexture [Example]
• Set the MTLTexture to the contents of the plane's material

[CheshireCat92]

@rsfuller , thx for answer. I'll try to use your advices and write if this solution will work in project.

[Cloov]

Hi,

I'm experiencing one of the problems originally stated by Cheshire Cat:

After all of this, the fox mask rendered so big. It looks like my face's geometry 3x bigger that it's real.

I need to record using a Portrait camera preset, and when I do this, I need to follow one of two approaches:

• Rotate the pixel buffer before sending it to the face session update method
• Instead, modify the recognition rotation that goes into the face session update method, and also the call to projectionMatrixForViewportSize

The second approach is better as it doesn't require an expensive pixel buffer modification, and that's what I'm trying to use, but there's an issue with the size of the fox mask.

I see that the projectionMatrixForViewportSize has the following comment:

This method assumes camera images are rendered to aspect fill, not aspect fit.

Is there a way for this to be modified so that it works for aspect fit?

I still feel like the face mesh looks so much bigger than a change from aspect fit to aspect fill might not account for as much difference in the face scale, but I thought I'd bring up the comment too.

Any help is appreciated, thanks.

[ntqp97]

Hi,

I'm experiencing one of the problems originally stated by Cheshire Cat:

After all of this, the fox mask rendered so big. It looks like my face's geometry 3x bigger that it's real.

I need to record using a Portrait camera preset, and when I do this, I need to follow one of two approaches:

• Rotate the pixel buffer before sending it to the face session update method
• Instead, modify the recognition rotation that goes into the face session update method, and also the call to projectionMatrixForViewportSize

The second approach is better as it doesn't require an expensive pixel buffer modification, and that's what I'm trying to use, but there's an issue with the size of the fox mask.

I see that the projectionMatrixForViewportSize has the following comment:

This method assumes camera images are rendered to aspect fill, not aspect fit.

Is there a way for this to be modified so that it works for aspect fit?

I still feel like the face mesh looks so much bigger than a change from aspect fit to aspect fill might not account for as much difference in the face scale, but I thought I'd bring up the comment too.

Any help is appreciated, thanks.

I am also facing the same problem as above. Has anyone got a solution to this problem?