Bounds of faces detected are offset when using it with Skia frame processors

[srikanth-c-p]

Using skia to draw bounding boxes around the detected face, but the boxes are always offset to the bottom right.

const { detectFaces } = useFaceDetector({ performanceMode: 'fast', contourMode: 'none', landmarkMode: 'none', classificationMode: 'none', autoScale: false, });

`const frameProcessor = useSkiaFrameProcessor((frame) => { 'worklet'; const faces = detectFaces(frame); frame.render();

const canvasWidth = frame.width;
const canvasHeight = frame.height;

// Centralized oval bounds for face fitting
const ovalBounds = {
  x: canvasWidth / 4,
  y: canvasHeight / 4,
  width: canvasWidth / 2,
  height: canvasHeight / 2,
};

// Calculate oval center
const ovalCenterX = ovalBounds.x + ovalBounds.width / 2;
const ovalCenterY = ovalBounds.y + ovalBounds.height / 2;

// Margin of error for detection
const marginOfError = 50; // Adjust as needed for user comfort

let faceIsWithinBounds = false;

if (faces.length > 0 && !isProcessing) {
  faces.forEach((face) => {
    // Scale the face coordinates to match the canvas dimensions
    const scaleX = canvasWidth / frame.width;
    const scaleY = canvasHeight / frame.height;

    const faceX = face.bounds.x * scaleX;
    const faceY = face.bounds.y * scaleY;
    const faceWidth = face.bounds.width * scaleX;
    const faceHeight = face.bounds.height * scaleY;

    // Draw the bounding box around the face
    const paint = Skia.Paint();
    paint.setColor(Skia.Color('blue')); // Bounding box color (blue for debugging)
    paint.setStyle(PaintStyle.Stroke);
    paint.setStrokeWidth(5);
    frame.drawRect({ x: faceX, y: faceY, width: faceWidth, height: faceHeight }, paint);

    // Check if face center is within the oval bounds with a margin of error
    const faceCenterX = faceX + faceWidth / 2;
    const faceCenterY = faceY + faceHeight / 2;

    faceIsWithinBounds =
      Math.abs(faceCenterX - ovalCenterX) <= marginOfError &&
      Math.abs(faceCenterY - ovalCenterY) <= marginOfError;
  });

  if (faceIsWithinBounds) {
    myFunctionJS(); // Triggers JS function to set the face detected state
  }
}

// Draw the static oval with updated color based on detection
const ovalPaint = Skia.Paint();
ovalPaint.setColor(Skia.Color(faceIsWithinBounds ? 'green' : 'red')); // Green if face detected within bounds
ovalPaint.setStyle(PaintStyle.Stroke);
ovalPaint.setStrokeWidth(5);
frame.drawOval(ovalBounds, ovalPaint); // Draw the static oval

}, []); ` Expected behaviour: there is a static oval on screen that is supposed to change color when a face is detected in it. This version of the code is with some scaling attempted on the actual bounds but it hasn't worked. (the blue box is the one drawn based on the bounds recieved.) I can hardcode an offset to make it work with my device, but need a solution that should work regardless of the device being used.

Device: Samsung Galaxy M51 IOS: Android

package.json:

{
  "name": "face",
  "version": "0.0.1",
  "private": true,
  "scripts": {
    "android": "react-native run-android",
    "ios": "react-native run-ios",
    "lint": "eslint .",
    "start": "react-native start",
    "test": "jest"
  },
  "dependencies": {
    "@shopify/react-native-skia": "^1.3.13",
    "axios": "^1.7.7",
    "react": "18.3.1",
    "react-native": "0.75.2",
    "react-native-fs": "^2.20.0",
    "react-native-reanimated": "^3.15.1",
    "react-native-vision-camera": "^4.5.3",
    "react-native-vision-camera-face-detector": "^1.7.1",
    "react-native-worklets-core": "^1.3.3"
  },
  "devDependencies": {
    "@babel/core": "^7.20.0",
    "@babel/preset-env": "^7.20.0",
    "@babel/runtime": "^7.20.0",
    "@react-native/babel-preset": "0.75.2",
    "@react-native/eslint-config": "0.75.2",
    "@react-native/metro-config": "0.75.2",
    "@react-native/typescript-config": "0.75.2",
    "@types/react": "^18.2.6",
    "@types/react-test-renderer": "^18.0.0",
    "babel-jest": "^29.6.3",
    "eslint": "^8.19.0",
    "jest": "^29.6.3",
    "prettier": "2.8.8",
    "react-test-renderer": "18.3.1",
    "typescript": "5.0.4"
  },
  "engines": {
    "node": ">=18"
  }

screenshot attached to better convey the issue

[luicfrr]

Try setting your blue box x position at const faceX = face.bounds.x - face.bounds.width;. Also, there's no need to use scale with skia as it uses frame size to work and your scale is always 1 anyway

const canvasWidth = frame.width;
...
const scaleX = canvasWidth / frame.width; // you're dividing frame.width by frame.width - makes no sense

[Boukhtam]

For me I've and a fter observing frame width and height and face bounds width and height which make no sense to me! successfully calculated "actualFaceBounds" as follows:

// I don't even know what to call these variable names though 
const scaleX = layout.width / frame.width;
const scaleY = layout.height / frame.height;

const actualFaceBounds = {
  x: face.bounds.x * scaleX,
  y: face.bounds.y,
  width: face.bounds.width,
  height: face.bounds.height * scaleY,
}

@luicfrr here is a log of some values:

console.log({ frameWidth: frame.width, frameHeight: frame.height }) 
// outputs: {"frameHeight": 480, "frameWidth": 640}
console.log({ faceBoundsWidth: face.bounds.width, faceBoundsHeight: face.bounds.height })
// outputs: {"faceBoundsHeight": 220, "faceBoundsWidth": 217}

while the app is in portrait mode, and face bounds seems almost like a square! Any explanations?

[lutfiarahmanda]

for me i need to rotate it to 90 degree as frame was rotated

      // Center the frame
      frame.translate(frame.width / 2, frame.height / 2);
      frame.rotate(90, 0, 0); // Adjust rotation based on orientation
      frame.translate(-frame.height / 2, -frame.width / 2);

      const faces = detectFaces(frame);
      
                const rect = Skia.XYWHRect(
            frame.height - face.bounds.x - face.bounds.height,
            face.bounds.y,
            face.bounds.width,
            face.bounds.height,
          );

[luicfrr]

I added some examples using skia in example app and it's working well. Can you guys test it before I release it as a new version?

[luicfrr]

fixed on c1cce1b