🐛 In landscape mode, enabling frameProcessor causes preview image stretching and incorrect orientation

[boiboif]

What's happening?

In landscape mode

Preview without frameProcessor:

Preview with frameProcessor enabled:

Reproduceable Code

const frameProcessor = useFrameProcessor(
    frame => {
      'worklet';
    },
    [handleDetectedFaces],
  );

<Camera
  ref={camera}
  style={StyleSheet.absoluteFill}
  device={device}
  isActive={true}
  photo
  // frameProcessor={frameProcessor}
/>

Relevant log output

> Configure project :react-native-vision-camera
[VisionCamera] Thank you for using VisionCamera ??
[VisionCamera] If you enjoy using VisionCamera, please consider sponsoring this project: https://github.com/sponsors/mrousavy
[VisionCamera] node_modules found at C:\Users\87332\Desktop\tc\healthybd-app\node_modules
[VisionCamera] VisionCamera_enableFrameProcessors is set to true!
[VisionCamera] react-native-worklets-core found, Frame Processors are enabled!
[VisionCamera] VisionCamera_enableCodeScanner is set to false!

Camera Device

{
  "formats": [],
  "sensorOrientation": "landscape-left",
  "hardwareLevel": "full",
  "maxZoom": 10,
  "minZoom": 1,
  "maxExposure": 24,
  "supportsLowLightBoost": false,
  "neutralZoom": 1,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "supportsFocus": true,
  "supportsRawCapture": false,
  "isMultiCam": false,
  "minFocusDistance": 10,
  "minExposure": -24,
  "name": "0 (BACK) androidx.camera.camera2",
  "hasFlash": true,
  "hasTorch": true,
  "position": "back",
  "id": "0"
}

Device

redmi k50

VisionCamera Version

4.5.3

Can you reproduce this issue in the VisionCamera Example app?

No, I cannot reproduce the issue in the Example app

Additional information

• [ ] I am using Expo
• [ ] I have enabled Frame Processors (react-native-worklets-core)
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar issues in this repository and found none.

[maintenance-hans[bot]]

Guten Tag, Hans here! 🍻

I see you have provided some information about ze issue, but to help mrousavy resolve it effectively, we need to gather more logs. Please provide logs from adb logcat while reproducing ze issue, as they can help diagnose what’s happening during runtime.

Also, it might be worth considering sponsoring ze project if you’d like more immediate attention on this issue: Sponsor mrousavy.

Let’s make sure ze issue is clear so we can get to ze bottom of it together!

Note: If you think I made a mistake, please ping @mrousavy to take a look.

[meleffendi]

I'm having the same issue on 4.5.2 on OnePlus5.

In my case, when the phone is in portrait mode, the preview shows correct. If I rotate the phone to landscape, I get the incorrect orientation and stretching.

Were you able to fix it?

[boiboif]

I'm having the same issue on 4.5.2 on OnePlus5.我在 OnePlus5 的 4.5.2 上遇到了同样的问题。

In my case, when the phone is in portrait mode, the preview shows correct. If I rotate the phone to landscape, I get the incorrect orientation and stretching.就我而言，当手机处于纵向模式时，预览显示正确。如果我将手机旋转到横向，我会得到不正确的方向和拉伸。

Were you able to fix it?你能修好它吗？

My application is a landscape-oriented app. Currently, my solution is to force the camera screen to portrait orientation.

const portraitHoc = (Component: React.ComponentType<any>) => {
  return (props: any) => {
    const [hasLocked, setHasLocked] = React.useState(false);
    const previousOri = React.useRef<OrientationType>();

    React.useEffect(() => {
      let timer: NodeJS.Timeout;
      locker.getOrientation(o => {
        previousOri.current = o;
        locker.lockToPortrait();
        timer = setInterval(() => {
          locker.getOrientation(o => {
            if (o === OrientationType.PORTRAIT) {
              setHasLocked(true);
              clearInterval(timer);
            }
          });
        }, 100);
      });

      return () => {
        clearInterval(timer);
        if (previousOri.current === OrientationType['LANDSCAPE-RIGHT']) {
          locker.lockToLandscapeRight();
        } else if (previousOri.current === OrientationType['LANDSCAPE-LEFT']) {
          locker.lockToLandscapeLeft();
        } else {
          locker.lockToLandscape();
        }
      };
    }, []);

    return hasLocked ? <Component {...props} /> : null;
  };
};

export default portraitHoc(Facedetect);

[dalisalvador]

This problem can be reproduced using the example app. When using frameProcessor, the Frame.orientation is fixed to the native orientation of the device. What I don't understand is why this has to also lock the Preview.

Related to https://github.com/mrousavy/react-native-vision-camera/issues/3051#issuecomment-2243708666

[charles474]

@mrousavy Just wondering if there is any update regarding this open bug?

[Shubham0850]

I'm also facing the same issue. Any update @mrousavy

[Shubham0850]

@charles474 did you able to find anything yet ?

[arnnis]

I'm having the same issue on 4.6.4, Only happens when frame processor is active in landscape mode

[eebirke]

I think it has something to do with texture and surface views. When vision-camera passes the current orientation to the preview you can see an error in LogCat about rotation not being supported on SurfaceView.PERFORMANCE or something like that. When I'm testing with unlocked app rotation, it seems to behave as expected if I set the androidPreviewType prop on the CameraView to "texture-view" (when switching between the two I usually have to do a full reload of the app/component)

The docs describe possible differences between the two here https://react-native-vision-camera.com/docs/api/interfaces/CameraProps#androidpreviewviewtype

"surface-view": Uses a SurfaceView for rendering. This is more efficient and supports HDR rendering, but doesn't support masks, transparency, rotations or clipping. "texture-view": Uses a TextureView for rendering. This is less efficient and doesn't support HDR rendering, but supports masks, transparency, rotations and clipping.

A more pressing issue in my case is that it seems like the Image/ImageProxy itself cannot be rotated by MLKit – haven't tested it yet, but that might also break CodeScanner in some phone orientations with certain sensor orientations 🤔 #3490

[charles474]

@charles474 did you able to find anything yet ?

@Shubham0850 Sadly, I haven't.

[mingca]

@dalisalvador Were you able to fix this issue?

[eduardorfreitas93]

https://github.com/mrousavy/react-native-vision-camera/issues/3051#issuecomment-3010647771