mrousavy
feat: Add `depth` to `Frame`
What
TL;DR: Adds Frame.depth, a nullable CMSampleBuffer that holds depth data.
If enableDepthData is true (a prop passed to <Camera>), the Camera will be configured to set up a depth data output pipeline in addition to the video data output pipeline.
Ontop of that, a synchronizer output is created to sync depth data and video data buffers together for timestamps.
const device = Camera.getAvailableCameraDevices().find((d) => d.formats.some((f) => f.supportsDepthCapture))
const format = useCameraFormat(device, [{ depth: true }])
const fp = useFrameProcessor((frame) => {
'worklet'
console.log(frame.depth)
})
if (device == null) throw new Error(`Device does not support depth data!`)
return (
<Camera
device={device}
format={format}
enableDepthData={true}
video={true}
frameProcessor={fp}
/>
)
Changes
Tested on
Related issues
The latest updates on your projects. Learn more about Vercel for Git ↗︎
| Name | Status | Preview | Comments | Updated (UTC) |
|---|---|---|---|---|
| react-native-vision-camera | ✅ Ready (Inspect) | Visit Preview | 💬 Add feedback | Feb 14, 2025 3:08pm |
![vercel[bot] avatar](https://avatars.githubusercontent.com/in/8329?v=4 "Published by a pub.dev verified publisher"){kind=small}
merge this shit pls.
Did you even bother to try this PR yourself to confirm everything works as expected?
merge this shit pls.
Did you even bother to try this PR yourself to confirm everything works as expected?
I sincerely apologize for my previous comment. The frustration took over my keyboard.
Yes, I did test it locally and confirmed it was not working, I successfully fixed the dept locally as I needed it urgently. I’m more than happy to contribute so this can get merged.
This PR is not yet working as expected because it will stall the video pipeline. So it'll not be merged yet, until that's fixed. But I got lots of other stuff to do rn
@mrousavy Hi. This is an amazing feature. Would be great if it also allows support for android. Is it possible to integrate ARCore with Raw Depth API or Depth API? Thank you.
@rologr35 CameraX doesn't support depth data streaming. And any code I'd write in manual Camera2 bridging code would be a workaround, I don't want to release that stuff to production.
@mrousavy First, thank you for working on this depth data feature! I'm developing an app that requires LiDAR depth data for 3D measurements. While I wait for this PR to be merged, would it also be possible to expose depth data in the PhotoFile interface when taking photos with enableDepthData=true? This would be incredibly valuable for capturing high-quality depth maps without processing every frame.
@mrousavy We are very excited for this feature! I'm just curious if there's a timeline on its release?
