🐛 Can't position square representing detected QR code frame properly

[antoinerousseau]

What's happening?

On my test phone (an iPhone XR), I managed to position it quite well:

but on my colleague's phone (iPhone 14 pro) it's quite off:

Reproduceable Code

// the relevant code of onCodeScanned:
if (code.frame) {
  // TODO: handle device.sensorOrientation (here it's assumed "landscape")

  const top = code.frame.x / WINDOW.scale - cameraViewSizes.y;
  const height = Math.max(100, code.frame.height / WINDOW.scale);

  const right = code.frame.y / WINDOW.scale;
  const width = Math.max(100, code.frame.width / WINDOW.scale);

  squareBounds.value = {
    right: right - (width / cameraViewSizes.width) * right,
    width,
    top: top - (height / cameraViewSizes.height) * top,
    height,
  };
}

// the relevant JSX:
<TouchableWithoutFeedback
  onPress={onPressCamera}
  onPressIn={onHoldCamera}
  onPressOut={onReleaseCamera}
>
  <View style={[styles.main, style]} onLayout={onCameraWrapperLayout}>
    <VisionCamera
      ref={cameraRef}
      torch={flashMode}
      style={styles.scanner}
      device={device}
      isActive={!disableCamera}
      codeScanner={codeScanner}
      enableZoomGesture
    />

    <Reanimated.View style={animatedStyle}>
      <Text style={styles.holdText}>Hold to scan</Text>
    </Reanimated.View>

    // ... & some more absolute views here for controls
  </View>
</TouchableWithoutFeedback>

// the relevant styles:
const styles = StyleSheet.create({
  main: {
    position: "relative",
    backgroundColor: "#050505",
  },
  scanner: {
    flex: 1,
    backgroundColor: "#050505",
  },
});

Relevant log output

frame {"height": 190.0366759300232, "width": 181.95819854736328, "x": 1022.731819152832, "y": 289.8436903953552}
squareBounds {"height": 100, "right": 110.41120430697566, "top": 368.3117007502803, "width": 100}

Camera Device

for my iPhone XR:

{
  "hardwareLevel": "full",
  "isMultiCam": false,
  "supportsLowLightBoost": false,
  "maxExposure": 8,
  "sensorOrientation": "landscape-right",
  "neutralZoom": 1,
  "supportsFocus": true,
  "supportsRawCapture": false,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "minExposure": -8,
  "name": "Back Camera",
  "hasFlash": true,
  "id": "com.apple.avfoundation.avcapturedevice.built-in_video:0",
  "maxZoom": 16,
  "minZoom": 1,
  "position": "back",
  "hasTorch": true
}

Device

my colleague's: iPhone 14 Pro

VisionCamera Version

3.6.12

Can you reproduce this issue in the VisionCamera Example app?

I didn't try (⚠️ your issue might get ignored & closed if you don't try this)

Additional information

• [ ] I am using Expo
• [ ] I have enabled Frame Processors (react-native-worklets-core)
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar issues in this repository and found none.

[nobodyzzz01]

any update ?

[DevUser0491]

Hey @antoinerousseau Any update? Please, could you share your code?

[mrousavy]

Hey - the CodeScanner now gives you a frame/bounding box as well, hope that helps.

[antoinerousseau]

@mrousavy do you mean #2117? If so, it was released in v3.6.5 and this issue is with v3.6.12, and the second params just gives the width/height of the scanning area

[mrousavy]

Yes exactly - using that you can figure out where exactly the Frame was scanned in, then convert that to screen coordinates yourself - no?

[dann1609]

@antoinerousseau any fix? I'm facing same issue. Looks like frame data is wrong

[daankennes]

I updated from 4.0.4 to 4.3.2 today and run into the same issue. Prior to the version update I had to recalculate the coordinates, but they were correct. My code in 4.0.4 (with some minor changes):

function onCodeScanned(codes: Code[], frame: CodeScannerFrame) {
    const temporaryCornerPointsSets: CornerPointsSets = [];
    codes.forEach((code) => {
        if (!code || !code.corners || !code.frame || !code.value) return;

        /*
         * Returned code coordinates are on different coordinate system than UI coordinate
         * system (frame coordinates vs device-independent pixels). Recalculate them:
         */
        // height and width reversed, since frame is (always?) landscaped
        const frameHeight = Math.max(frame.height, frame.width);
        const frameWidth = Math.min(frame.height, frame.width);
        const transformationX = dimensions.width / frameWidth;
        const transformationY = dimensions.height / frameHeight;

        // todo: rewrite with .map
        const point0X = code.corners[0].x * transformationX;
        const point0Y = code.corners[0].y * transformationY;

        const point1X = code.corners[1].x * transformationX;
        const point1Y = code.corners[1].y * transformationY;

        const point2X = code.corners[2].x * transformationX;
        const point2Y = code.corners[2].y * transformationY;

        const point3X = code.corners[3].x * transformationX;
        const point3Y = code.corners[3].y * transformationY;

        const translatedCornerPoints = [
            {x: point0X, y: point0Y},
            {x: point1X, y: point1Y},
            {x: point2X, y: point2Y},
            {x: point3X, y: point3Y},
        ];

        if (translatedCornerPoints.every(isCoordinateInFinder)) {
            temporaryCornerPointsSets.push(translatedCornerPoints);
        }
    });

    if (temporaryCornerPointsSets.length === 0) {
        setCornerPointsSets(undefined);
    } else if (temporaryCornerPointsSets.length === 1) {
        setCornerPointsSets(temporaryCornerPointsSets);
    } else {
        setCornerPointsSets(temporaryCornerPointsSets);
    }

}

Now the coordinates just don't make sense anymore. Has anything changed?

[daankennes]

My problem appears to be the default resizeMode (which is cover). Because of this, codes are also scanned outside the preview which caused the incorrect mapping. I now use resizeMode contain and that seems to fix it. I don't understand how it worked before.