💭 Enabling `video` slows down photo capture on Android

[jkaufman]

Question

Photo capture performance drops significantly and results in a preview layer freeze when the video prop is set to true.

Condition	Format Filter	Photo	Video	Duration[^1]
0	undefined	true	false	373.5ms
1	[{ photoResolution: { width: 480, height: 320 } }]	true	false	389.6ms
2	undefined	true	true	779.7ms
3	[{ photoResolution: { width: 480, height: 320 } }]	true	true	823.8ms

[^1]: Averaged across 10 sequential captures.

What I tried

Test app: rnvc3-test.zip

VisionCamera Version

3.8.2

Additional information

• [X] I am using Expo
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar questions in the issues page as well as in the discussions page and found none.

[mrousavy]

Can you give a bit more insights, did you test this on Android or iOS? Which Android phone did you test this on? Is this also reproduceable on iOS?

[mrousavy]

Ah, I have an idea! - can you try something please?

https://github.com/mrousavy/react-native-vision-camera/blob/7e2889cf84c0346fda8589ae69d56706bad85efd/package/android/src/main/java/com/mrousavy/camera/core/CameraSession.kt#L579

After this line, add:

captureSession.stopRepeating()

This will break the preview, but I can fix that later if this works.

Try to see if that makes things faster :)

[jkaufman]

Confirmed. That brings Condition 2 in the table down to 218.5ms. All testing performed on a Pixel 3a.

[mrousavy]

Perfect, then I know how to speed that up! I'll leave this open for now until I implement it.

[mrousavy]

Heyo, I just tested this on my end on my Huawei P10:

Without the stopRepeating() before capture, so just VisionCamera main codebase as it is today:

14:09:55.089 CameraSession            I  CAPTURE() took 265ms!
14:09:57.138 CameraSession            I  CAPTURE() took 232ms!
14:09:59.106 CameraSession            I  CAPTURE() took 249ms!
14:10:01.044 CameraSession            I  CAPTURE() took 241ms!
14:10:02.768 CameraSession            I  CAPTURE() took 241ms!
14:10:04.496 CameraSession            I  CAPTURE() took 225ms!
14:10:06.293 CameraSession            I  CAPTURE() took 263ms!
14:10:12.641 CameraSession            I  CAPTURE() took 254ms!

With the stopRepeating() added before the capture, so with the patch I suggested:

14:09:08.248 CameraSession            I  CAPTURE() took 260ms!
14:09:11.473 CameraSession            I  CAPTURE() took 246ms!
14:09:13.598 CameraSession            I  CAPTURE() took 250ms!
14:09:15.459 CameraSession            I  CAPTURE() took 233ms!
14:09:17.609 CameraSession            I  CAPTURE() took 254ms!
14:09:19.516 CameraSession            I  CAPTURE() took 234ms!
14:09:21.597 CameraSession            I  CAPTURE() took 246ms!

There doesn't seem to be a difference here at all on my end.

Can you absolutely confirm that this changes the capture speed for you?

[mrousavy]

Created a PR for this based off of current main, test if you can: https://github.com/mrousavy/react-native-vision-camera/pull/2517

[jkaufman]

Thanks. I don't have the Pixel 3a, right now, but will test once I do.

[jkaufman]

Still working to reproduce. I encounter an Expo plugin error attempting to install from the test branch.

[jkaufman]

app.plugin.js appears to point to a missing lib directory

[mrousavy]

Run yarn.

[mrousavy]

Hey - did you try the PR? Would be great if you could check if that makes any difference in 3.9.0.

[jkaufman]

Yes, thanks for checking in. I encountered some unrelated difficulties running locally, but got it working just now. Looks good to me.

Condition	Format Filter	Photo	Video	Duration
0	undefined	true	false	247.2ms
1	[{ photoResolution: { width: 480, height: 320 } }]	true	false	170.8ms
2	undefined	true	true	264.4ms
3	[{ photoResolution: { width: 480, height: 320 } }]	true	true	227.6ms

[mrousavy]

Condition 0 and 1 is on main, and Condition 2 and 3 is with that PR I assume?

I updated the PR a few hours ago, did you pull latest off that branch ?

[jkaufman]

Those are all measurements from your fix branch. Compare it to the table in my original report to contrast it with main. I pulled it just before commenting. On main, capture duration was slower with video enabled. This table shows that your PR mitigates the slowdown.

[mrousavy]

Ah I see - but I landed a bunch of improvements in main since you created that PR, any chance you could benchmark that again on main? Just so we have reference values if the PR really improves things.

[jkaufman]

I'm having difficulty building it, again. I've set the branch in my package.json, run yarn and yarn build in the /package directory for the library in node_modules. Still getting an error that the plugin cannot be resolved.

Edit: Simple error. I was referencing the project root rather than the /package directory.

[jkaufman]

3.9.0 produces very different and counterintuitive behavior.

With no parameters passed to takePhoto:

Condition	Format Filter	Photo	Video	Duration
0	undefined	true	false	2100ms
1	skipped
2	undefined	true	true	TIME OUT
3	skipped

With {qualityPrioritization: 'speed'} passed to takePhoto:

Condition	Format Filter	Photo	Video	Duration
0	undefined	true	false	1100ms
1	skipped
2	undefined	true	true	300ms
3	skipped

[mrousavy]

Yea, I didn't know there were so many different vendors/manufacturers not properly implementing Camera HAL for AE/AF/AWB. On all my test devices it worked perfectly fine. I guess this needs more workarounds I am just not sure what kinds of workarounds.

[jkaufman]

Could performance on Pixel devices be a clue? I'd expect them to be a reference implementation of the Camera HAL. Perhaps it makes sense to target them, first.

[mrousavy]

Hey! I just released a new V4 beta (v4.0.0-beta.7) where I fixed a bunch of issues! 💪🚀 Can you test that and see if that fixes the issue for you? 😅

yarn add react-native-vision-camera@beta

You might need to increase the compileSdk to the newest Android SDK (34) if you are behind.

[jkaufman]

Results for 4.0.0-beta.8:

Condition	Format Filter	Photo	Video	Duration
0	undefined	true	false	921ms
1	skipped
2	undefined	true	true	697ms
3	skipped

To recap, capture with only photo enabled is slower in beta-8 than capture with video also enabled. Both conditions are slower than either of the same conditions in 3.8.2.

[jkaufman]

Results for 4.0.0-beta.12:

Condition	Format Filter	Photo	Video	Duration
0	undefined	true	false	535ms
1	skipped
2	undefined	true	true	729ms
3	skipped

[mrousavy]

Yea I didn't change anything about that yet. But thanks for re-testing.

[mrousavy]

wait nvm it looks like the values flipped? maybe this is a bit more random than I thought? In V4 we always to the auto-exposure precapture routine (AE + AF + AWB) so this might just take some unavoidable time? takeSnapshot() is for insanely fast capture

[jkaufman]

Yes, it flipped, and there was a lot of variation in timings, although video true was consistently slower. I also saw strings of 900ms captures with the device face up on the table and no light reaching the back camera.

Mind sharing what to expect from takeSnapshot of the preview layer? Are AE / AF / AWB disabled, or is it polling and periodically updating the preview layer, accordingly?

[mrousavy]

It's just reading the latest pixels from the preview or video stream. No AE/AF/AWB at all.

[Zeronfinity]

After this line, add:

captureSession.stopRepeating()

This will break the preview, but I can fix that later if this works.

Just curious, what kind of fix do you have in mind to keep Preview smooth despite stopRepeating?

A stopRepeating followed by a startRepeating has chance of recreating the capture session and may cause some freeze time (or temporary black screen) in Preview (depends on camera HAL implementation, so may vary across devices).

By the way, one possible root cause for this bug may actually be session reconfiguration. If you change a session parameter while submitting a non-repeating capture request for taking still picture, the camera may need to reconfigure the capture session to accommodate that (again, depends on HAL implementation so will vary across devices). And the tricky part is - if you are setting a session parameter on repeating request and not setting it to non-repeating request, default value from template will be used when unset and that may differ from your repeating request.

For example, imagine you have 60 FPS set to repeating request for video capture and not setting the FPS in non-repeating request which is using TEMPLATE_STILL_REQUEST, the FPS will change to 30 FPS since that's usually the default value for this template. Since 30 and 60 FPS usually require different sensors, the session may be reconfigured in some devices which may be the reason of the latency.

One way to debug this could be by logging all the capture request keys and values you are submitting to the camera framework and check if any different value is being sent when the bug is happening.

You can also try using the camera watch command for this. For example, with your device connected via adb, run a command like adb shell cmd media.camera watch start -m android.control.captureIntent,android.flash.mode,3a && adb shell cmd media.camera watch live on a terminal with the app turned on and then capture an image and see the differences in the capture tags (or just copy-paste here). (note that this command is monitoring only the 3a-related tags mainly, but you can add other tags as well if the library is using them)

Oh also, as far as I remember, this library is migrating to CameraX as well, right? Does the problem occur with CameraX as well?

[mrousavy]

Hey @Zeronfinity - that's really interesting, I also didn't know about the Camera Debugging section.

Seems like you really know your sh*t about cameras, are you using VisionCamera or did you just chime in out of interest? :)

[mrousavy]

And yea, react-native-vision-camera 4.x.x is using CameraX now - I'm sure you are aware of all the quirks Camera2 had, and how the entire CameraX team is dedicated to fix or workaround these device-specific quirks/bugs.

There's no point in using Camera2, I don't stand a chance against hundreds of device-specific bugs that I need to workaround myself. Better off to leave the Google engineers work on this for CameraX, especially now that CameraX also has stream-sharing and HDR features.

Only thing that I noticed was that the Frame Processors (CameraX ImageAnalysis) is much slower than my Camera2 ImageReader implementation. I thought about creating a custom output that streams HardwareBuffers using OpenGL textures to skip the ImageReader step, that way I could also do the RGB conversion straight on the GPU.