Add depth data processing for iOS (TrueDepth and dual camera)

[thomas-coldwell]

What

• This PR adds a AVCaptureDepthDataOutput which enables the depth data to be passed to frame processor plugins.
• It will also introduce a new prop enableDepthData to select whether depth data should be included when using one of the depth enabled cameras e.g. builtInDualCamera or builtInTrueDepth

Changes

• Adds a AVCaptureDepthDataOutput
• Adds a AVCaptureDataOutputSynchronizer to sync up the video and depth data
• Adds the corresponding delegate class for the synchronizer and handles the processing in the method dataOutputSynchronizer rather than captureOutput as it is not a single output
• Adds a depth property of type CVPixelBuffer? to the Frame class

Tested on

• iPhone 13 Pro, 15.1.1

[thomas-coldwell]

I've edited a bunch of stuff just to test out the depth data in frame processors that I'll remove in a new commit as well as linting + formatting btw. Not ready to merge yet 😅

[mrousavy]

Shouldn't depth also be a Frame?

So the TypeScript type should look like this:

// Already existing Frame type
type Frame = {
  width: number
  height: number
  ...
  depth: Omit<Frame, 'depth'>
}

no? Because the depth frame can actually have a different width/height than the RGB/YUV frame.

Edit

Technically that's already how it works, but in the JSI HostObject we just need to return a new Frame HostObject when the user is accessing .depth, I can do that too.

[thomas-coldwell]

Yeah it will need to be its own frame with depth as a property of that. I haven't updated that end of it yet - purely checking if the depth buffer comes through on the frame processor at the minute, but will need to add that as a property of it.

[thomas-coldwell]

The last commit just adds depthDataFormat as a prop now for the Camera component so that the user can specify which format they want given they have selected a format that contains their desired depth data format.

Also the depth can be toggled on/off with enableDepthData which simply switches it to just configure the video output as before.

[thomas-coldwell]

Took a stab at it and seems to all be working - omitted isValid and close() from the depth frame too as these only make sense in the CMSampleBuffer context, but the depth is just the raw CVPixelBuffer.

I think the last thing to look at is just making sure the AVCaptureOutputSynchronizer is properly syncing the video and depth outputs. Once this is good we can do some tests in a frame processor just to check its getting a stable depth value (at least from pixel to pixel, if not absolute physical accuracy in meters) and then I think its about there 😃 If there's anything else you spot that I've missed or we want to pass into the frame processor callback let me know

[OlleThunbergDevoteam]

What's the status of this PR?

Is it functional?

[mrousavy]

This PR is amazing work, but unfortunately got a ton of conflicts because of V3.

Let's keep this open for reference if I implement depth at some point. The synchronizer is amazing!

I wonder if this is possible on Android at all?