How do you handle Pose Landmarker Orientation Changes on Android and iOS?

[frank-fsj]

Hi all, I am using the Pose Landmarker in my Android and iOS projects that support Portrait, Landscape Left, Landscape Right orientations with camera live streaming mode. I am following along the Google guidelines in which my objective is to use the coordinates to create angles. In both Android and iOS, I have already made the orientation adjustments so that both the camera and pose overlay appear in the proper orientation when the device changes orientation.

The problem I am facing is that even when the orientation is in landscape, the imageAnalyzer is still processing everything in portrait, and so I am getting different joint angles when I change between portrait and landscape orientations.

Android CameraFragment.kt: val cameraSelector = CameraSelector.Builder().requireLensFacing(cameraFacing).build()

val resolutionSelector = ResolutionSelector.Builder() .setAspectRatioStrategy( AspectRatioStrategy(AspectRatio.RATIO_4_3, AspectRatioStrategy.FALLBACK_RULE_NONE) ) .build()

val displayRotation = fragmentCameraBinding.viewFinder.display?.rotation ?: Surface.ROTATION_0

preview = Preview.Builder() .setTargetRotation(displayRotation) .setResolutionSelector(resolutionSelector) .build()

imageAnalyzer = ImageAnalysis.Builder().setTargetAspectRatio(AspectRatio.RATIO_4_3) .setTargetRotation(fragmentCameraBinding.viewFinder.display.rotation) .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST) .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888) .build()

    .also {
        it.setAnalyzer(backgroundExecutor) { image ->
            detectPose(image)
        }
    }

In portrait, AspectRatio.RATIO_4_3 is correct, but as soon as you change to landscape, it still remains in portrait AspectRatio.RATIO_4_3, it never actually changes based upon the current rotation. Even though the points of the human skeleton for the Pose Overlay look to be correct, the coordinates are using a portrait 4:3.

As a test only: val originalX = landmark.x() val originalY = landmark.y()

val newX = originalX * (4f / 3f) val newY = originalY * (3f / 4f)

I converted the x, y for coordinates from portrait to landscape based upon the 4:3 ratio, and the angles are now correct in both landscape and portrait, but I don’t think this is a good method to handle this. Can you please recommend a better way to do this?

iOS: As the Mediapipe example is older now, there are a lot of places that require orientation updates as some of the previous orientation change methods have been deprecated for iOS 17.0+. func detectAsync( sampleBuffer: CMSampleBuffer, orientation: UIImage.Orientation, timeStamps: Int) { guard let image = try? MPImage(sampleBuffer: sampleBuffer, orientation: orientation) else { return } do { try poseLandmarker?.detectAsync(image: image, timestampInMilliseconds: timeStamps) } catch { print(error) } }

I am not sure how to convert between the portrait and landscape orientations as I did not find any aspect ratios listed for the iOS version to use such as like in Android. Am I supposed to use detectAsync here? I did try 4:3 and 16:9, but it didn’t seem to create the same angles for both portrait and landscape. I assume it is just using the screen width and height, but I am not sure, and I am less confident to use screen width and height with such different sizes between iphones and ipad so I am hoping to get some insight on how to properly handle orientation changes. Thank you.