react-native-vision-camera 🐛 Frame processor is getting distorted, whereas inbuild code scanner is working fine

What's happening?

I have to process the barcode and OCR both in my application therefore I am relying on frame processor but the frame processor returns distorted UI, even without using any other plugin. When I use the builtIn barcode scanner the video is not distorted but then for the OCR I have to rely on taking snapshots and processing that instead.

There is a major difference between built in frame processor and code scanner in all android devices. This is from frame processor Media (1) This when using code scanner

Reproduceable Code

const frameProcessor = useFrameProcessor((frame) => {
		'worklet';

		console.log(frame.height);
	}, []);

	const codeScanner = useCodeScanner({
		codeTypes: ['qr', 'ean-13'],
		onCodeScanned: (codes) => {
			'worklet';

			console.log(`Scanned ${codes.length} codes!`);
		},
	});

Relevant log output

12-20 15:06:34.738 31241 31879 I CameraView: invokeOnAverageFpsChanged(19.95798319327731)
12-20 15:06:34.772 31241 31430 I ReactNativeJS: 1080
12-20 15:06:34.776   449 31942 E awb_adpt: 2336, awb_sprd_ctrl_calculation_v3: check handle success
12-20 15:06:34.776   449 31936 E smart_ctrl: 1911, smart_ctl_calc_atm: jhin_atm atm_switch_state:0 1
12-20 15:06:34.776   449 31936 E smart_ctrl: 1914, smart_ctl_calc_atm: jhin_atm atm_version 2 
12-20 15:06:34.776   449 31936 D ATM     : 1099, atm_calc: smooth out frame_id 70 u4EVD 256 smooth u4EVD 256.000000  bv -266
12-20 15:06:34.779   449 31924 I Libae[HM]: 1073,hm:cam:ae,WEIGHT IS -267,0,6,15

Camera Device

{
  "hardwareLevel": "limited",
  "minExposure": -32,
  "neutralZoom": 1,
  "minZoom": 1,
  "supportsFocus": true,
  "formats": [],
  "supportsLowLightBoost": false,
  "hasTorch": true,
  "supportsRawCapture": false,
  "minFocusDistance": 0.09775171065493646,
  "sensorOrientation": "landscape-left",
  "maxZoom": 4,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "isMultiCam": false,
  "position": "back",
  "id": "0",
  "hasFlash": true,
  "maxExposure": 32,
  "name": "0 (BACK) androidx.camera.camera2"
}

Device

narzo 50i (Android)

VisionCamera Version

4.5.1/4.6.3

Can you reproduce this issue in the VisionCamera Example app?

Yes, I can reproduce the same issue in the Example app here

Additional information

[X] I am using Expo
[X] I have enabled Frame Processors (react-native-worklets-core)
[X] I have read the Troubleshooting Guide
[X] I agree to follow this project's Code of Conduct
[X] I searched for similar issues in this repository and found none.

Dec 20 '24 09:12 Swikriti0998

Guten Tag, Hans here! 🍻

Thank you for providing detailed information about your issue with the frame processor. It seems there might be an actual bug or difference in how the processor is handling the frames compared to the built-in code scanner.

However, I'd like to ask you to double-check your logs for any additional clues during the distortion of the frame. You can gather more logs by using adb logcat while reproducing the issue, as it will help mrousavy in diagnosing the problem.

Your logs are a bit sparse, and having more context would greatly aid in troubleshooting.

If you're enjoying the library, consider supporting the project through sponsorship. This helps keep development going and ensures faster responses to issues like yours!

Looking forward to more details!

Note: If you think I made a mistake, please ping @mrousavy to take a look.

Dec 20 '24 09:12 maintenance-hans[bot]

Guten Tag, Hans here! 🍻

Thanks for providing ze detailed issue description, issue logs, and reproducible code. It looks like you have made a good effort in explaining what’s happening.

However, to help mrousavy understand ze issue better, could you please check if there are any additional logs related to ze frame processor? You can gather logs using adb logcat while reproducing ze issue, as it can reveal more information about what might be going wrong.

In ze meantime, it may also help to ensure your camera configuration is optimized for both barcode scanning and OCR processing. If you can gather more logs or provide insights on your setup, it would be really helpful!

Feel free to reach out with more info!

Note: If you think I made a mistake, please ping @mrousavy to take a look.

Dec 20 '24 09:12 maintenance-hans[bot]

Apparently the issue is with the camera format if you use the format key in the camera component is causes the frame to give distorted image even if you are using the current resolution.

<Camera device={device} isActive={true} format={format} // this is causing distorted video with frame processor. onError={handleError} style={{ height: cameraHeight, width }} />

Dec 20 '24 10:12 Swikriti0998

Apparently the issue is with the camera format if you use the format key in the camera component is causes the frame to give distorted image even if you are using the current resolution.

<Camera device={device} isActive={true} format={format} // this is causing distorted video with frame processor. onError={handleError} style={{ height: cameraHeight, width }} />

I can confirm the same is happening. I wanted to limit barcode scanning FPS but it requires passing the format prop, which even though is set at the current best, results in a very low quality preview.

Device tested on- Samsung Galaxy M13 & Emulator using a 1080p webcam as back camera.

Using the below versions (all latest as of now):

RNVC - 4.6.3 react-native-worklets-core - 1.5.0 react-native-reanimated - 3.16.6

Jan 07 '25 09:01 gyanbhartip

@mrousavy, can you please take a look at this and let us know if we're missing anything? With the format, the reading is faster, but the video appears distorted. It’s working fine on iOS, though.

Feb 07 '25 10:02 Swikriti0998

Noticing the same issue on android when just adding/removing the format created from useCameraFormat, even with it set to max res. On the emulator it seems to be working correctly, but on the real device the image gets very blurry/pixelated.