react-native-ml-kit face detection returns []

Hello, I'm facing an issue with my IOS device. When I send an image URL captured through a react-native-vision-camera it returns an empty array. Do you know how I can solve this?

const photo = await camera.current.takePhoto({ enableShutterSound: false, });

try { const faceResults = await FaceDetection.detect('file://' + photo.path, { contourMode: 'all', }); console.log(faceResults); } catch (e) { console.log(e); }

EX: I didn't use any firebase package.

Oct 19 '23 12:10 vidurajith-darshana

Hi @vidurajith-darshana! Thanks for reporting this issue. I'll check it and get back to you once I have something to share 🙏🏼.

Oct 19 '23 17:10 a7medev

@a7medev thanks for the reply. Hope some quick solution as soon as possible. we are stuck with this issue. Please note I've never used firebase in my app and if there are any additional installation guides will be really helpful 🙏🏼.

Oct 19 '23 17:10 vidurajith-darshana

Hi everyone! I have same issue but only with ios devices and selfie photos made by them. If I download selfie photo from internet it works). I have same issue in react-native-face-detection module too. I use react-native-image-picker module and its methods: launchImageLibrary and launchCamera to get a selfie. If I share problem selfie somewhere and then download it then this selfie works) magic! It looks like ios protects reading images made by its device.

Update: Same issue is reproduced in your example too

iOS 16.5.1 (20F75) iPad (5th generation) (Model A1822)

Oct 24 '23 21:10 zheoreh

@vidurajith-darshana

I've never used firebase in my app and if there are any additional installation guides will be really helpful 🙏🏼.

you can setup firebase with https://rnfirebase.io/ if you are using expo here is how https://docs.expo.dev/guides/using-firebase/#using-react-native-firebase Do not use the Firebase JS SDK

Oct 26 '23 17:10 flexbox

@flexbox Actually he asked does he needs to setup Firebase for react-native-ml-kit ? he asked if there were any other installation instructions related to react-native-ml-kit based on Firebase, it's helpful. not about the Firebase setup

Oct 29 '23 13:10 senturatechnologies

Hi everyone! I have same issue but only with ios devices and selfie photos made by them. If I download selfie photo from internet it works). I have same issue in react-native-face-detection module too. I use react-native-image-picker module and its methods: launchImageLibrary and launchCamera to get a selfie. If I share problem selfie somewhere and then download it then this selfie works) magic! It looks like ios protects reading images made by its device.

Update: Same issue is reproduced in your example too

iOS 16.5.1 (20F75) iPad (5th generation) (Model A1822)

Same on me

Oct 31 '23 09:10 nth-zik

@a7medev is there any update, please? we have been stuck with this issue since last week.

Oct 31 '23 12:10 vidurajith-darshana

@vidurajith-darshana have you found some solution?

Nov 13 '23 17:11 zheoreh

no luck, still the same

Nov 14 '23 01:11 vidurajith-darshana

after debugging, i think the problem is with the temp file path from vision-camera output. As the photo file is stored in a temporary location, that is not accessible directly by file path. It cannot work with that file path atm.

Dec 07 '23 16:12 codemem

it's throwing error like “ABA4B2D6-A241-4DF6-A1BA-FD6F984A2EE3-1864-0000024DF9AE4C71.jpg” couldn’t be opened because there is no such file."

Dec 07 '23 16:12 codemem

seems we don't have an answer here. I've been struggling for weeks here using this library. any PR is welcome for the issues. This is going to be an inactive, not-maintained repo.

Dec 07 '23 19:12 vidurajith-darshana

I will try to see if i can open pr. Take a rest :)

Dec 07 '23 21:12 codemem

great! but did it give your results as you expected? when I try to get the face detection results, it always returns []. did you have your results correctly?

Dec 08 '23 14:12 vidurajith-darshana

hmm sorry, i messed up the lib. i supposed to comment on other issue from vision-camera-face-detection not here. No it's still not working

Dec 08 '23 14:12 codemem

it does not work with vision-camera. I think there is smt with temp file. It works fine with local photo from library but not with temp file name file:///private/blah when working with local file it returns correct result. file path from photo library file:///var/mobile

[{"frame": {"height": 1206, "left": 530, "top": 968, "width": 1206}, "rotationX": -5.255302906036377, "rotationY": -3.0993707180023193, "rotationZ": -1.612923502922058}] faces

p/s: on IOS

Dec 13 '23 22:12 codemem

I had similar issue with another library https://github.com/teslamotors/react-native-camera-kit/issues/568

For me, its not to do with file location as suggested but to do with image orientation. Here is how I patched mine so that I can make this work for the react-native-camera-kit library I am using. I could be wrong but I don't think this has to do with the mlkit itself.

Create a patch file (lookup react native patch if you are not familiar with this) patches/@react-native-ml-kit+face-detection+1.3.1.patch
Paste these file contents

index 87fd4ff..f237b71 100644
--- a/node_modules/@react-native-ml-kit/face-detection/ios/FaceDetection.m
+++ b/node_modules/@react-native-ml-kit/face-detection/ios/FaceDetection.m
@@ -234,6 +234,15 @@ - (NSDictionary*)faceToDict: (MLKFace*)face
     return dict;
 }
 
+- (UIImage*)fixImageOrientation: (UIImage*)image
+  {
+    UIGraphicsBeginImageContext(image.size);
+    [image drawAtPoint:CGPointZero];
+    UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext();
+    UIGraphicsEndImageContext();
+    return newImage ?: image;
+}
+
 RCT_EXPORT_METHOD(detect: (nonnull NSString*)url
                   withOptions: (NSDictionary*)optionsDict
                   resolver:(RCTPromiseResolveBlock)resolve
@@ -241,7 +250,9 @@ - (NSDictionary*)faceToDict: (MLKFace*)face
 {
     NSURL *_url = [NSURL URLWithString:url];
     NSData *imageData = [NSData dataWithContentsOfURL:_url];
-    UIImage *image = [UIImage imageWithData:imageData];
+    UIImage *imageraw = [UIImage imageWithData:imageData];
+    UIImage *image = [self fixImageOrientation:imageraw];
+
     MLKVisionImage *visionImage = [[MLKVisionImage alloc] initWithImage:image];
     MLKFaceDetectorOptions *options = [self getOptions:optionsDict];
     MLKFaceDetector *faceDetector = [MLKFaceDetector faceDetectorWithOptions:options];

Jun 14 '24 15:06 write2sv

` diff --git a/node_modules/@react-native-ml-kit/face-detection/RNMLKitFaceDetection.podspec b/node_modules/@react-native-ml-kit/face-detection/RNMLKitFaceDetection.podspec index 0db7d5f..366aef5 100644 --- a/node_modules/@react-native-ml-kit/face-detection/RNMLKitFaceDetection.podspec +++ b/node_modules/@react-native-ml-kit/face-detection/RNMLKitFaceDetection.podspec @@ -21,6 +21,6 @@ Pod::Spec.new do |s| s.requires_arc = true

s.dependency "React"

s.dependency "GoogleMLKit/FaceDetection", "4.0.0"

s.dependency "GoogleMLKit/FaceDetection", "5.0.0" end

diff --git a/node_modules/@react-native-ml-kit/face-detection/android/.gitignore b/node_modules/@react-native-ml-kit/face-detection/android/.gitignore deleted file mode 100644 index 6125af2..0000000 --- a/node_modules/@react-native-ml-kit/face-detection/android/.gitignore +++ /dev/null @@ -1,4 +0,0 @@ -.iml -gradle/ -gradlew -gradlew.bat diff --git a/node_modules/@react-native-ml-kit/face-detection/ios/FaceDetection.m b/node_modules/@react-native-ml-kit/face-detection/ios/FaceDetection.m index 87fd4ff..909e178 100644 --- a/node_modules/@react-native-ml-kit/face-detection/ios/FaceDetection.m +++ b/node_modules/@react-native-ml-kit/face-detection/ios/FaceDetection.m @@ -234,15 +234,23 @@ - (NSDictionary)faceToDict: (MLKFace*)face return dict; }

-RCT_EXPORT_METHOD(detect: (nonnull NSString*)url

             withOptions: (NSDictionary*)optionsDict

+RCT_EXPORT_METHOD(detect:(NSString * _Nonnull)url

             withOptions:(NSDictionary *)optionsDict
             resolver:(RCTPromiseResolveBlock)resolve
             rejecter:(RCTPromiseRejectBlock)reject)

{ NSURL *_url = [NSURL URLWithString:url]; NSData *imageData = [NSData dataWithContentsOfURL:_url];

UIImage *image = [UIImage imageWithData:imageData];
MLKVisionImage *visionImage = [[MLKVisionImage alloc] initWithImage:image];

UIImage *originalImage = [UIImage imageWithData:imageData];
// EXIF 정보를 고려하여 이미지 회전
UIImage *fixedImage = [self fixImageOrientation:originalImage];
// EXIF 없는 새 이미지 데이터 생성
NSData *fixedImageData = UIImageJPEGRepresentation(fixedImage, 1.0);
UIImage *imageForDetection = [UIImage imageWithData:fixedImageData];
MLKVisionImage *visionImage = [[MLKVisionImage alloc] initWithImage:imageForDetection]; MLKFaceDetectorOptions options = [self getOptions:optionsDict]; MLKFaceDetector faceDetector = [MLKFaceDetector faceDetectorWithOptions:options]; [faceDetector processImage:visionImage @@ -260,4 +268,77 @@ - (NSDictionary)faceToDict: (MLKFace)face }]; }

+// EXIF 정보를 고려하여 이미지 회전 +- (UIImage *)fixImageOrientation:(UIImage *)image {

if (image.imageOrientation == UIImageOrientationUp) return image;
CGAffineTransform transform = CGAffineTransformIdentity;
switch (image.imageOrientation) {
```
   case UIImageOrientationDown:
```

   case UIImageOrientationDownMirrored:

       transform = CGAffineTransformTranslate(transform, image.size.width, image.size.height);

       transform = CGAffineTransformRotate(transform, M_PI);

```
       break;
```
```
   case UIImageOrientationLeft:
```

   case UIImageOrientationLeftMirrored:

       transform = CGAffineTransformTranslate(transform, image.size.width, 0);

       transform = CGAffineTransformRotate(transform, M_PI_2);

```
       break;
```
```
   case UIImageOrientationRight:
```

   case UIImageOrientationRightMirrored:

       transform = CGAffineTransformTranslate(transform, 0, image.size.height);

       transform = CGAffineTransformRotate(transform, -M_PI_2);

```
       break;
```
```
   case UIImageOrientationUp:
```
```
   case UIImageOrientationUpMirrored:
```
```
       break;
```
}
switch (image.imageOrientation) {
```
   case UIImageOrientationUpMirrored:
```

   case UIImageOrientationDownMirrored:

       transform = CGAffineTransformTranslate(transform, image.size.width, 0);

       transform = CGAffineTransformScale(transform, -1, 1);

```
       break;
```

   case UIImageOrientationLeftMirrored:

   case UIImageOrientationRightMirrored:

       transform = CGAffineTransformTranslate(transform, image.size.height, 0);

       transform = CGAffineTransformScale(transform, -1, 1);

```
       break;
```
```
   case UIImageOrientationUp:
```
```
   case UIImageOrientationDown:
```
```
   case UIImageOrientationLeft:
```
```
   case UIImageOrientationRight:
```
```
       break;
```
}
CGContextRef ctx = CGBitmapContextCreate(NULL, image.size.width, image.size.height,

                                        CGImageGetBitsPerComponent(image.CGImage), 0,

                                        CGImageGetColorSpace(image.CGImage),

                                        CGImageGetBitmapInfo(image.CGImage));

CGContextConcatCTM(ctx, transform);
switch (image.imageOrientation) {
```
   case UIImageOrientationLeft:
```

   case UIImageOrientationLeftMirrored:

```
   case UIImageOrientationRight:
```

   case UIImageOrientationRightMirrored:

       CGContextDrawImage(ctx, CGRectMake(0,0,image.size.height,image.size.width), image.CGImage);

```
       break;
```
```
   default:
```

       CGContextDrawImage(ctx, CGRectMake(0,0,image.size.width,image.size.height), image.CGImage);

```
       break;
```
}
CGImageRef cgimg = CGBitmapContextCreateImage(ctx);
UIImage *img = [UIImage imageWithCGImage:cgimg];
CGContextRelease(ctx);
CGImageRelease(cgimg);
return img; +}

@end

`

It is working

just do remove the exif information ...

Jul 29 '24 16:07 SlZeroth