Using the model with rotations

[chapmacl]

I am looking at the example and it works just fine in portrait mode. I understand that it defaults to 90 since android's camera captures photos in landscape by default, but what does changing the value actually do?

More specifically, I am trying to get this to work with an ImageStream in different orientations. Landscape seems to not work whatsoever. I would assume I would need to set it to 0 or 180, which I have tried, but still no luck.

Any idea on why this wouldn't work by simply adjusting rotation? Do I also need to switch height/width?

[chapmacl]

After fiddling around with it, I have managed to get the model to work with rotating (as well as many headaches with translating the bounding boxes). I understand that 'rotation' simply transposes the matrix of the image byte data, and gives coordinates relative to the rotated photo as output.

I am still struggling, however, with how the model performs during rotation. In portrait (both 0 and 180) the model is very accurate and detects many objects, but as soon as I go to landscape (90 and 270), the model fails to recognize all of the objects it had seconds ago been able to detect, and sometimes cannot detect certain objects at all.

Is there any explanation why the model struggles so much with rotated photos?

[chapmacl]

I am locking the camerapreview in portrait and measuring the rotation of the device, then passing the degrees of rotation (0, 90, 180, 270) along with the rotation of the sensor (for android this is often 90). Anything I am missing?

      Tflite.detectObjectOnFrame(
        bytesList: img.planes.map((plane) {
          return plane.bytes;
        }).toList(),
        model: "SSDMobileNet",
        imageHeight: img.height,
        imageWidth: img.width,
        imageMean: 127.5,
        imageStd: 127.5,
        threshold: 0.6,
        numResultsPerClass: 5,
        rotation:
            rotation + controller.description.sensorOrientation,`