Question - Capture depth and rgb by arkit current frame

[giovil]

Hello,

first of all I would like to congratulate you because your plugin is really well done.

Currently, for several reasons, I should acquire rgb and depth map frames no longer via AVCaptureSession but through the use of ARKit. I am currently trying to convert your plugin to start an ARSession and get the frames by accessing the buffers of the current frame.

I start the session correctly, I created the delegate to have the callback event in Unity, but when I try to access the buffer for the construction of the texture2d it gives me the following error "No texture data provided to LoadRawTextureData". Have you had this problem before?

Would you be interested in taking a look at the code or help me?

[AtsushiSuzuki]

I don't have any experience with ARKit. Could you show me actual code?

[giovil]

Here the xCode and Unity projects: https://we.tl/t-rVPHoDGLOo

Thanks a lot!!

[AtsushiSuzuki]

It seems that pVideoData is NULL or width * height * 4 is 0 in ARKitController.cs. see https://github.com/Unity-Technologies/UnityCsReference/blob/d09d3f534a99b5367eba548f634ef3ac4079fcf2/Runtime/Export/Graphics/Texture.cs#L616 Texture2D.LoadRawTextureData requires valid data pointer and size.

Currently width = videoWidth and height = videoHeight is commented out so size == 0. Can you uncomment them and try?

[giovil]

Thank you, it was a little oversight. Now, after solving the empty RawTextureData, I have the problem of how the image looks.

As you can see it seems in B&W (in the native side the image is colored and appears correctly) and is splitted into multiple views and only the first one in the upper left is updated.

Do you have any idea?

[AtsushiSuzuki]

Next, you need to make sure source image format (pixel format and resolution) is same as destination image format. You are using ARFaceTrackingConfiguration.supportedVideoFormats.last but you need to choose apropriate image resolution. ARKit document says pixel format is YUV (https://developer.apple.com/documentation/arkit/arframe/2867984-capturedimage) so TextureFormat.BGRA32 is not compatible. TextureFormat.YUY2 may be compatible, but I'm not sure.