fix: Add support for YUV_420_888 Image format.

[panmari]

Casting the bytes to a type directly is not possible, thus allocating a new texture is necessary (and costly).

But on the bright side, we avoid the conversion inside the camera plugin [0].

[0] https://github.com/flutter/packages/blob/d1fd6232ec33cd5a25aa762e605c494afced812f/packages/camera/camera_android/android/src/main/java/io/flutter/plugins/camera/media/ImageStreamReaderUtils.java#L35

[panmari]

There's still the open issue of cleaning up the texture. I'd appreciate some feedback on how to change the API accordingly. I don't think there's a way around that.

[wantroba]

If you allow me, I will share with you the method I used to solve a similar problem. Maybe it can help:

import 'dart:io';
import 'dart:isolate';
import 'dart:typed_data';
import 'package:camera/camera.dart';
import 'package:image/image.dart' as imglib;
import 'package:path_provider/path_provider.dart';
import 'package:flutter/material.dart';

class ImageUtils {
  static Future<imglib.Image?> convertToImage(CameraImage image) async {
    if (image.format.group == ImageFormatGroup.yuv420) {
      return convertYUV420toImageColor(image);
    } else {
      return imglib.Image.fromBytes(
        bytes: image.planes[0].bytes.buffer,
        width: image.width,
        height: image.height,
        order: imglib.ChannelOrder.bgra,
      );
    }
  }

  static Future<imglib.Image?> convertYUV420toImageColor(
      CameraImage image) async {
    try {
      final int width = image.width;
      final int height = image.height;
      final int uvRowStride = image.planes[1].bytesPerRow;
      final int uvPixelStride = image.planes[1].bytesPerPixel ?? 0;

      debugPrint("uvRowStride: $uvRowStride");
      debugPrint("uvPixelStride: $uvPixelStride");

      ReceivePort port = ReceivePort();
      final isolate = await Isolate.spawn<SendPort>((sendPort) {
        // imgLib -> Image package from https://pub.dartlang.org/packages/image
        // Create Image buffer
        // Fill image buffer with plane[0] from YUV420_888
        var img = imglib.Image(width: width, height: height);
        for (int x = 0; x < width; x++) {
          for (int y = 0; y < height; y++) {
            final int uvIndex =
                uvPixelStride * (x / 2).floor() + uvRowStride * (y / 2).floor();
            final int index = y * width + x;

            final yp = image.planes[0].bytes[index];
            final up = image.planes[1].bytes[uvIndex];
            final vp = image.planes[2].bytes[uvIndex];
            // Calculate pixel color
            int r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255);
            int g = (yp - up * 46549 / 131072 + 44 - vp * 93604 / 131072 + 91)
                .round()
                .clamp(0, 255);
            int b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255);
            // color: 0x FF  FF  FF  FF
            //           A   B   G   R
            img.data?.setPixel(x, y, imglib.ColorRgb8(r, g, b));
          }
        }
        sendPort.send(img);
      }, port.sendPort);
      imglib.Image img = await port.first;
      isolate.kill(priority: Isolate.immediate);
      return img;
    } catch (e) {
      debugPrint(">>>>>>>>>>>> ERROR:$e");
    }
    return null;
  }

  static Future<File?> createFile(imglib.Image image) async {
    imglib.Command cmd = imglib.Command()
      ..image(image)
      ..encodePng();
    await cmd.executeThread();
    Uint8List? imageBytes = cmd.outputBytes;
    if (imageBytes != null) {
      cmd = imglib.Command()..decodePng(imageBytes);
      await cmd.executeThread();
      imglib.Image? originalImage = cmd.outputImage;
      int height = originalImage?.height ?? 0;
      int width = originalImage?.width ?? 0;
      var dir = await getApplicationDocumentsDirectory();
      File file = File(
          "${dir.path}/help_buy_app_${DateTime.now().millisecondsSinceEpoch}.png");
      // Let's check for the image size
      if (height >= width) {
        // I'm interested in portrait photos so
        // I'll just return here
        cmd = imglib.Command()
          ..decodePng(imageBytes)
          ..writeToFile(file.path);
        await cmd.executeThread();
        return file;
      }

      if (height < width && originalImage != null) {
        debugPrint('Rotating image necessary');
        cmd = imglib.Command()
          ..image(originalImage)
          ..copyRotate(angle: 90)
          ..writeToFile(file.path);
        await cmd.executeThread();
        return file;
      }
    }

    return null;
  }
}

[panmari]

I fixed the resource leak by making InputImageConverter implement AutoClosable and calling the close function after processing is done.

@wantroba thanks for providing an alternative solution! I feel it targets a slightly different use case (saving to file) as the one I target here (using as ML-Kit input).

[panmari]

May I ask what was the reason for closing this PR? It did add an additional feature, which allows clients to move to the better supported camera_android_camerax package.

[fbernaly]

my mistake, could you resolve the conflicts.

[fbernaly]

could you resolve the conflicts

[FantaMagier]

@panmari It would be great if you could resolve the conflicts. That would be great.

[panmari]

Rebased and resolved conflicts. The caveat that the conversion code is expensive still applies.

[FantaMagier]

Hey any news on this important rp? @fbernaly @panmari Would be great!

[panmari]

@fbernaly this is ready for review.

[AncientPixel]

Is there any movement on this, we have production issues that we need to resolve and waiting another month... would not be nice.

[panmari]

If you're consuming images directly from the camera package, there's always the option to the alternative implementation to consume the supported image format, see https://github.com/flutter-ml/google_ml_kit_flutter/blob/d430f0232464a1744c4c05e8e49396ef7c9a9d35/packages/example/pubspec.yaml#L15