✨ Implement Orientation ($8,000)

[mrousavy]

trafficstars

What feature or enhancement are you suggesting?

Currently, there is no direct orientation support for VisionCamera.

This feature request contains of multiple parts:

• Support orientation prop: When this is passed, the Camera will rotate itself to the target orientation
• Take Camera Sensor orientation (often 90deg) into consideration for relative orientation
• Save correct orientation in captured photos (relative to sensor and relative to device/orientation prop)
• Save correct orientation in recorded videos (relative to sensor and relative to device/orientation prop)
• Pass correct orientation in Frame in Frame Processor (relative to sensor and relative to device/orientation prop)
• Documentation on how to properly handle orientation (see iOS Camera app, the camera itself should not rotate, only controls should rotate)

This comes with a ton of research, development and testing on both iOS and Android so it's quite complex to handle properly.

Also, fun fact: On Android this is even more annoying. Check out this blog post - Samsung exposes orientation wrong, meaning I have to add special handling for such devices. Insane.

The Android documentation is quite okay though.

$8,000

I'll implement Orientation for $8,000. Donate to the polar pool to support the feature.

[valn1]

what should the orientation prop do in this case? i imagine the default "none" to just do nothing, "auto" to read the device's orientation and adjust accordingly and "portrait/landscape" to use that aspect ratio. but i wonder how it should respond to that prop changing while recording a video

[mrousavy]

Yea good point. I think those should be the options:

• "portrait" | "portraitUpsideDown": Locked at portrait orientation.
• "landscapeLeft" | "landscapeRight": Locked at landscape orientation.
• "device": Use phone orientation and always rotate when phone rotates. This is not really recommended as it requires Camera reconfiguration.
• undefined -> "portrait"

[valn1]

cool, should that not be changeable mid recording?

[mrousavy]

No, a recording is started at a specific orientation and will always stay at that orientation. Orientation in a recording is just an EXIF flag. it has a fixed width/height.

[valn1]

not as of right now. i fixed that in another app of mine that runs the v2 by using react-native-orientation-locker and doing a patch on the ios part of react-native-vision-camera, but i dont think it works for pictures or video as i used it only for barcode scanning

[tiste]

I really needed to implement orientation quickly, so here is a small hack to make it work on both landscape right and left (as it works on portrait).

<View
  style={{
    transform: [
      isLandscapeRight ? { rotate: "-90deg" } : { rotate: "90deg" },
    ],
    width: layout.height,
    height: layout.width,
    position: "absolute",
    left: layout.width / 2 - layout.height / 2,
    top: layout.height / 2 - layout.width / 2,
  }}
>
  <Camera
    enableHighQualityPhotos={true}
    orientation={
      isLandscapeRight ? "landscape-right" : "landscape-left"
    }
    device={device}
    isActive={true}
    ref={cameraRef}
    style={{
      width: layout.height,
      height: layout.width,
    }}
    video={true}
    photo={true}
  />
</View>

[mrousavy]

not as of right now

@valn1 I know, I'm talking about this proposal here.

[mrousavy]

Related: https://github.com/mrousavy/react-native-vision-camera/issues/2046

[erickcrus]

Im using @oguzhnatly/react-native-image-manipulator package to handle image rotation after take a picture when necessary. This is not ideal but works.

For example:

import RNImageManipulator from '@oguzhnatly/react-native-image-manipulator';

photo = await cameraRef.current?.takePhoto();
const device = await getDeviceInfo();          // internal function to get device information

if (['Samsung', 'samsung'].includes(device.manufacturer)) {
    // rotate to right position when using samsung front camera
    const rotated = await RNImageManipulator.manipulate(photo.path, [{ rotate: photo.isMirrored ? 90 : 0 }], { compress: 1, format: 'jpeg' }); 
    const output = rotated.uri;
}

[bglgwyng]

Actually, I haven't used orientation props since RNVC V2, as its change caused freeze on the view. So, I gave up utilizing orientation months ago and set it portrait. Also I implemented my frame processors to work well with explicitly provided orientation arguments, like

<Camera orientation="portrait" />

useFrameProcessor((frame) => {
  'worklet';
  scanFaces(frame, { orientation  })
});

I'm not entirely satisfied with this solution, but it works well, allowing the frame processors to adapt to any device orientation. While the non-functional orientation props can be worked around, the real issue lies in the incorrect frame.orientation. I'm really looking forward to the full-featured orientation props in RNVC V3. But before that, I hope a simple fix for frame.orientation can be merged.

Here I've created a branch to share the PoC of the fix. https://github.com/bglgwyng/react-native-vision-camera/tree/orientation-in-frame I thought the fix would be straightforwrad first, but now I'm unsure why the current isMirrored implementation is considered wrong as noted in the comment.

[bglgwyng]

frame.image remains unmirrored while the preview is mirrored on Galaxy S21. From my understanding, frame.image is expected to be mirrored, since mirrorMode is configured as MIRROR_MODE_AUTO. I set mirrorMode to MIRROR_MODE_NONE and observed that the preview changed unmirrored while frame.image didn't get affected. Does imageReader not respect the value of mirrorMode? Or is it another device-specific bug? It would be helpful if others with different devices could report similar behavior.

[mrousavy]

From my understanding, frame.image is expected to be mirrored, since mirrorMode is configured as MIRROR_MODE_AUTO

Not quite - actually mirroring an image is a computationally expensive task. Instead, the Camera uses special flags to tell the receiver (Preview View, Photo File .jpg, Video File, OpenGL pipeline, etc...) how to interpret the image. Only in rare cases the image is actually mirrored, but this will cause a latency.

So instead, we read the image in it's natural sensor orientation, then apply any orientations and mirroring handling ourselves once we save it for end user display (Preview View: view transform matrix, Photo File: exif flag, Video File: exif flag, OpenGL pipeline: well, actually doing a mirror here)

Same story on iOS, except that we curretly actually mirror/orientate the image - which we can also skip for better efficiency (see my idea at https://github.com/mrousavy/react-native-vision-camera/issues/2046)

[bglgwyng]

Sorry. My previous comment was a bit confusing. I also appreciate that frame.image contains the real-world image from the sensor. Though I wasn't aware of the performance issue, my focus was on the semantics and the convenience for ML tasks. I hope the current behavior on Android stays as is, and that the behavior on iOS, which currently mirrors the output, becomes the same as Android.

The statement 'frame.image is expected to be mirrored' doesn't convey a proposal but reflects my prediction. After reading some code of VideoPipeline, I had assumed that frame.image would be mirrored as it used the output configured as mirrorMode=MIRROR_MODE_AUTO. But it wasn't(on my Galaxy S21)! Maybe my understanding of the code is incorrect. Could you explain which part of the code keeps frame.image unmirrored?

[mrousavy]

The statement 'frame.image is expected to be mirrored' doesn't convey a proposal but reflects my prediction.

my man is speaking in cursive 😂

Well to be honest, we don't know if it is mirrored or not. Some devices might mirror it on the HAL level (Hardware Abstraction Layer), some won't. So I guess the ideal solution is to set MIRROR_MODE_NONE, and then handle mirroring on our end.

[ibmmt]

const photo = await cameraRef.current.takePhoto(); let rotation=0; if ( isFrontCamera && (photo.orientation == 'landscape-left' || photo.orientation == 'landscape-right')) { rotation = 90; } const photoWithOrientation = await manipulateAsync(photo.path, [{ rotate: rotation }], { compress: 1, format: SaveFormat.JPEG, });

Temporary solution.

[jazib10218]

I really needed to implement orientation quickly, so here is a small hack to make it work on both landscape right and left (as it works on portrait).

<View
  style={{
    transform: [
      isLandscapeRight ? { rotate: "-90deg" } : { rotate: "90deg" },
    ],
    width: layout.height,
    height: layout.width,
    position: "absolute",
    left: layout.width / 2 - layout.height / 2,
    top: layout.height / 2 - layout.width / 2,
  }}
>
  <Camera
    enableHighQualityPhotos={true}
    orientation={
      isLandscapeRight ? "landscape-right" : "landscape-left"
    }
    device={device}
    isActive={true}
    ref={cameraRef}
    style={{
      width: layout.height,
      height: layout.width,
    }}
    video={true}
    photo={true}
  />
</View>

Now camera is showing a zoomed in preview, how did you handle that ?

[haouarihk]

i needed a way to keep the resulting image in the same orientation soo..

i'm handling the image result orientation using this function

function orientationToRotationAngle(orientation: Orientation) {
    switch (orientation) {
        case 'portrait':
            return 90; // not tested
        case 'portrait-upside-down':
            return 180; // not tested
        case 'landscape-left':
            return 90; // tested
        case 'landscape-right':
            return 0; // tested
        default:
            // Handle unknown orientation
            return 0;
    }
}

and using a library called @bam.tech/react-native-image-resizer (btw you can't write @bam on windows shell, you need to add it to the package.json and then do npm i)

like so

const photo = await cameraRef.current?.takePhoto();

const uri = (await ImageResizer.createResizedImage(
    "file://" + photo?.path || ""
    , 1200, 720, "JPEG", 90, orientationToRotationAngle(device.sensorOrientation))
).uri

[mrousavy]

Related: https://stackoverflow.com/questions/48406497/

[mrousavy]

Also related: https://developer.android.com/codelabs/android-camera2-preview#5

[vanenshi]

This issue is causing problems with Google MLKit Face detection. here is the library I am trying to build vanenshi/vision-camera-face-detector

Technically MLKit needs the correct orientation to be able to detect the face, so when I manually set it to 90 deg, it works in portrait mode

[drLeonymous]

Hi, all! I'm not a Java/Kotlin friend but I did some research and here is a patch that works on the emulator Pixel_2_XL and Samsung Note10 and some other old Samsung or Huawei devices.

--- a/package/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt
+++ b/package/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt
@@ -14,6 +14,7 @@ import com.facebook.react.bridge.WritableMap
 import com.mrousavy.camera.core.CameraSession
 import com.mrousavy.camera.core.InsufficientStorageError
 import com.mrousavy.camera.types.Flash
+import com.mrousavy.camera.types.Orientation
 import com.mrousavy.camera.types.QualityPrioritization
 import com.mrousavy.camera.utils.*
 import java.io.File
@@ -46,7 +47,7 @@ suspend fun CameraView.takePhoto(optionsMap: ReadableMap): WritableMap {
     enableShutterSound,
     enableAutoStabilization,
     enablePrecapture,
-    orientation
+    cameraSession.orientation
  
+private val matrix = Matrix()
 private fun writePhotoToFile(photo: CameraSession.CapturedPhoto, file: File) {
   val byteBuffer = photo.image.planes[0].buffer
-  if (photo.isMirrored) {
     val imageBytes = ByteArray(byteBuffer.remaining()).apply { byteBuffer.get(this) }
     val bitmap = BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.size)
-    val matrix = Matrix()
-    matrix.preScale(-1f, 1f)
-    val processedBitmap =
-      Bitmap.createBitmap(bitmap, 0, 0, bitmap.width, bitmap.height, matrix, false)
+    matrix.reset()
+    if(photo.isMirrored) {
+      matrix.preScale(-1f, 1f)
+    }
+    matrix.postRotate(90f)
+    val processedBitmap = Bitmap.createBitmap(bitmap, 0, 0, bitmap.width, bitmap.height, matrix, false)
     FileOutputStream(file).use { stream ->
       processedBitmap.compress(Bitmap.CompressFormat.JPEG, 100, stream)
     }
-  } else {
-    val channel = FileOutputStream(file).channel
-    channel.write(byteBuffer)
-    channel.close()
-  }
 }
 
 private suspend fun savePhotoToFile(
diff --git a/package/android/src/main/java/com/mrousavy/camera/types/Orientation.kt b/package/android/src/main/java/com/mrousavy/camera/types/Orientation.kt
index 64dc5bcf..86275519 100644
--- a/package/android/src/main/java/com/mrousavy/camera/types/Orientation.kt
+++ b/package/android/src/main/java/com/mrousavy/camera/types/Orientation.kt
@@ -3,17 +3,17 @@ package com.mrousavy.camera.types
 import com.mrousavy.camera.core.CameraDeviceDetails
 
 enum class Orientation(override val unionValue: String) : JSUnionValue {
-  PORTRAIT("portrait"),
   LANDSCAPE_RIGHT("landscape-right"),
-  PORTRAIT_UPSIDE_DOWN("portrait-upside-down"),
-  LANDSCAPE_LEFT("landscape-left");
+  PORTRAIT("portrait"),
+  LANDSCAPE_LEFT("landscape-left"),
+  PORTRAIT_UPSIDE_DOWN("portrait-upside-down");
 
   fun toDegrees(): Int =
     when (this) {
-      PORTRAIT -> 0
-      LANDSCAPE_LEFT -> 90
-      PORTRAIT_UPSIDE_DOWN -> 180
-      LANDSCAPE_RIGHT -> 270
+        LANDSCAPE_LEFT -> 0
+        PORTRAIT -> 90
+        LANDSCAPE_RIGHT -> 180
+        PORTRAIT_UPSIDE_DOWN -> 270
     }
 
   fun toSensorRelativeOrientation(deviceDetails: CameraDeviceDetails): Orientation {
@@ -26,7 +26,7 @@ enum class Orientation(override val unionValue: String) : JSUnionValue {
     }
 
     // Rotate sensor rotation by target rotation
-    val newRotationDegrees = (deviceDetails.sensorOrientation.toDegrees() + rotationDegrees + 360) % 360
+    val newRotationDegrees = (deviceDetails.sensorOrientation.toDegrees() + rotationDegrees + 270) % 360
 
     return fromRotationDegrees(newRotationDegrees)
   }
@@ -34,19 +34,19 @@ enum class Orientation(override val unionValue: String) : JSUnionValue {
   companion object : JSUnionValue.Companion<Orientation> {
     override fun fromUnionValue(unionValue: String?): Orientation =
       when (unionValue) {
+        "landscape-left" -> LANDSCAPE_LEFT
         "portrait" -> PORTRAIT
         "landscape-right" -> LANDSCAPE_RIGHT
         "portrait-upside-down" -> PORTRAIT_UPSIDE_DOWN
-        "landscape-left" -> LANDSCAPE_LEFT
         else -> PORTRAIT
       }
 
     fun fromRotationDegrees(rotationDegrees: Int): Orientation =
       when (rotationDegrees) {
-        in 45..135 -> LANDSCAPE_LEFT
-        in 135..225 -> PORTRAIT_UPSIDE_DOWN
-        in 225..315 -> LANDSCAPE_RIGHT
-        else -> PORTRAIT
+          in 45..135 -> PORTRAIT
+          in 135..225 -> LANDSCAPE_RIGHT
+          in 225..315 -> PORTRAIT_UPSIDE_DOWN
+          else -> LANDSCAPE_LEFT
       }
   }
 }

[matheusqfql]

This is not working for me on Ipad with camera V3 (landscape mode)

[drLeonymous]

Use v4 beta 10 and rotate 90 deg the image when you save it on the backend server

[matheusqfql]

Use v4 beta 10 and rotate 90 deg the image when you save it on the backend server

The problem is that the live visualisation is 90 deg inverted on iPad landscape mode. I'm just using the barcode scanner (I'm not saving any image)

[simon-assistiq]

The problem is that the live visualisation is 90 deg inverted on iPad landscape mode. I'm just using the barcode scanner (I'm not saving any image)

Having the same issue as @matheusqfql. I'm a bit confused; is there no way to ensure the preview output of the camera on the screen is not rotated (I would need to rotate the preview by 90 degrees clockwise for it to be correct on the iPad in landscape) ? I'm migrating from 2.13.2 where the preview output was in the correct orientation, so I'm just wondering whether I'm just missing something new or some change was introduced. I don't need it to be able to respond to orientation changes, just want it to be always displayed in landscape on the iPad.

EDIT: Tried using orientation='landscape-right' on the <Camera> but it didn't seem to have any effect, which I guess is the point of this ticket. But the inline doc comment on that prop says that if this value isn't provided it should be using the device's orientation anyway.

I am also seeing this presumably related warning in the console:

[Orientation] BUG IN CLIENT OF UIKIT: Setting UIDevice.orientation is not supported. Please use UIWindowScene.requestGeometryUpdate(_:)

I have only tried iOS 16 so far, so I wonder if it would behave better on iOS 17.

[JesseLawler]

Does anyone have a good solution for this when applying to video?

Rotating an image is no big deal; I'm unsure of any reasonable solution for video...

[pvedula7]

I'm migrating from 2.13.2 where the preview output was in the correct orientation, so I'm just wondering whether I'm just missing something new or some change was introduced. I don't need it to be able to respond to orientation changes, just want it to be always displayed in landscape on the iPad.

Facing something somewhat similar where we are recording a video in standard portrait mode, but the frame.orientation always prints landscape-left. When attempting to overlay things onto the video with coordinates from frame processor, it's always off presumably due to some frame difference. Worked fine in V2 for us.

[mrousavy]

btw., I wrote an explanation on why rotation/orientation isn't as straight forward as it sounds here: https://github.com/mrousavy/react-native-vision-camera/pull/2807#issuecomment-2083079356

[mars-lan]

After reverting back to CameraX in 4.x, this doesn't seem to be an issue anymore.

[mrousavy]

This still is an issue - Frame Processors don't pass you a transformation matrix, and the preview view orientation cannot be frozen while outputs should rotate - that's how most Camera apps work.

Also, on iOS this still isn't implemented. For now, only portrait works perfectly.