InputImageRotation issues: Vertical images are not working, etc...

[bipin-sakariya]

In Flutter I am trying to detect text from images in horizontal orientation is working perfectly but Vertical orientation text is not detected. I am using the google_ml_kit plugin. Anyone has a solution for this or another way to achieve this functionality.

Here you can check it is working with native android :- https://github.com/googlesamples/mlkit/tree/master/android/vision-quickstart

[fbernaly]

@bipin-sakariya : with the camera or from the gallery? I guest it should be related to some orientation issue with your the image you are passing.

[prooshani]

Same Problem here.

I am using the example app provided by the package author. It works fine in portrait mode but if I rotate the device all the API's stop working or acting malfunction.

To be specific, I am using Samsung Galaxy Tab A7 SM-T505 right now and it is not working OK.

To test if there is a problem in imageOrientation I just added a log print over this function inside _processCameraImage:

...
final imageRotation = InputImageRotationValue.fromRawValue(camera.sensorOrientation) ?? InputImageRotation.rotation0deg;
    print('orientation: ${imageRotation.rawValue}');
...

and find that it is always 90 even if I rotate the device it will always detect my device in landscape mode. I don't know if I am testing and point in correct direction but I think it is something related to the cameraOrientation which brings this bug.

I am using the last version of camera and recent update of ml_kit.

google_ml_kit: ^0.10.0 camera: ^0.9.4+20

[fbernaly]

This issue has to do with the orientation of your device/camera/inputImage. The example app is not production code and only works on portrait mode. Feel free to modify the example app to support landscape, but you need to edit this method: https://github.com/bharat-biradar/Google-Ml-Kit-plugin/blob/master/packages/google_ml_kit/example/lib/vision_detector_views/camera_view.dart#L304-L342

I have not played that much with the orientation and metadata, but you need to create the correct instance of InputImagePlaneMetadata and InputImageData to pass to InputImage.fromBytes.

Once you figure it out come back and share the fix here or create a PR, contributions are always welcome.

[fbernaly]

Another comment in another discussion that might help you to resolve your issue is here: https://github.com/bharat-biradar/Google-Ml-Kit-plugin/issues/231#issuecomment-1113390253

[prooshani]

As you mentioned earlier, I believe it is something related to camera package which cannot find the correct aspect ratio after orientation change. To be exact, the imageProcess is working on something different than what you have in cameraPreview because, if you move the camera at the edge, where the barcode is merely outside the cameraPreview, you can see that it can detect the barcode correctly. SO, I think the image which is processing has different data than what we have in the cameraPreview.

I hope I could investigate further to see if I can match the zoom level of Landscape and Portrait in cameraPreview to see if the issue is resolved or not.

Any suggestion is highly appreciated.

[prooshani]

From my view the current culprit is this widget from camera_view.dart:

  Widget _liveFeedBody() {
    if (_controller?.value.isInitialized == false) {
      return Container();
    }

    final size = MediaQuery.of(context).size;
    // calculate scale depending on screen and camera ratios
    // this is actually size.aspectRatio / (1 / camera.aspectRatio)
    // because camera preview size is received as landscape
    // but we're calculating for portrait orientation
    var scale = size.aspectRatio * _controller.value.aspectRatio;

    // to prevent scaling down, invert the value
    if (scale < 1) scale = 1 / scale;

    return Container(
      width: 300,
      height: 400,
      color: Colors.black,
      child: Stack(
        fit: StackFit.expand,
        children: <Widget>[
         
          Transform.scale(
            scale: scale,
            child: Center(
              child: _changingCameraLens
                  ? Center(
                      child: const Text('Changing camera lens'),
                    )
                  : CameraPreview(_controller),
            ),
          ),
          if (widget.customPaint != null) widget.customPaint,
          Positioned(
            bottom: 100,
            left: 50,
            right: 50,
            child: Slider(
              value: zoomLevel,
              min: minZoomLevel,
              max: maxZoomLevel,
              onChanged: (newSliderValue) {
                setState(() {
                  zoomLevel = newSliderValue;
                  _controller.setZoomLevel(zoomLevel);
                });
              },
              divisions: (maxZoomLevel - 1).toInt() < 1 ? null : (maxZoomLevel - 1).toInt(),
            ),
          )
        ],
      ),
    );
  }

[prooshani]

Transform.scale( scale: scale, child: Center( child: _changingCameraLens ? Center( child: const Text('Changing camera lens'), ) : CameraPreview(_controller), ), ),

The landscape zoomed picture problem rooted from this part of the code:

 Transform.scale(
            scale: scale,
            child: Center(
              child: _changingCameraLens
                  ? Center(
                      child: const Text('Changing camera lens'),
                    )
                  : CameraPreview(_controller),
            ),
          ),

If you change the scale: scale to scale:1 it will fix the landscape orientation zoom problem but with some drawbacks. The main cons of doing this is the aspect ratio of the camera preview, which looks to be not the same as the main scaffold's body. After all, I think passing 1 to scale may solve the problem for now.