PNG file cannot be converted to Mat using OpenCvSharp4.runtime.wasm

[Aiprovide]

Summary of your issue

In Blazor WebAssembly, Mat.FromImageData is a png, jpg file, etc. and cannot be converted to Mat.

Environment

nuget: OpenCvSharp4 4.7.0.20230115 OpenCvSharp4.runtime.wasm 4.7.0.20230115

client (WebAssembly) Blazor in .NET 7

What did you do when you faced the problem?

In this sample, if Mandrill.bmp is converted to Mandrill.png, it cannot be converted to Mat correctly. https://github.com/shimat/opencvsharp_blazor_sample

Example code:

These codes did not work properly.

    var imageBytes = await httpClient.GetByteArrayAsync("images/Mandrill.png");
    srcMat ??= Mat.FromImageData(imageBytes);

or

public async void LoadFiles(InputFileChangeEventArgs e)
{
        long maxFileSize = 100 * 1024 * 1024; // 10MB
        var browserFile = e.File;
        using MemoryStream memoryStream = new MemoryStream();
        await browserFile.OpenReadStream(maxFileSize).CopyToAsync(memoryStream);
        srcMat = Mat.FromImageData(memoryStream.ToArray(), ImreadModes.Color);

Output:

What did you intend to be?

Please let me know if you have any solution. Thank you.

[Aiprovide]

Some BMP image files I own can be converted to Mat correctly, and some cannot. All PNG and JPG files could not be converted correctly.

[Aiprovide]

I know this is a WASM specific issue as it works fine in WPF.

[shimat]

https://github.com/shimat/opencvsharp/blob/bfd350fbb6219673d65bdc4c31a27ab035e6135d/.github/workflows/wasm.yml#L123

Unfortunately, in OpenCvSharp for wasm, the encoders/decoders for JPEG, PNG, and TIFF are disabled because linking with libpng, libtiff, etc. was not successful.

[Aiprovide]

Thank you for answering. Are there any plans to support PNG and JPG in WASM in the future? Is it difficult if libpng and libtiff cannot be linked?

[Aiprovide]

I tried Threshold, Canny and Akaze and they all worked with .NET7. libpng and libtiff may work with .NET7, but have you tried them?

[shimat]

I am developing opencvsharp5, an experimental project to rebuild OpenCvSharp for .NET 7, but even there, support for image libraries such as libpng and libtiff in wasm is failing. The cause of the failure is a linker error.

https://github.com/shimat/opencvsharp5/blob/30d61d3f55b4625722cf1df4c0e7ceead93d983f/.github/workflows/wasm.yml#L82

[LostBeard]

@Aiprovide Current workaround, in the browser, is to use an Image element to load the image, then draw it onto a canvas, read the canvas imageData, create a Mat of appropriate size and of the format CV_8UC4, and then write the image RGBA bytes to the Mat using Marshal.Copy. Quite a bit more effort than a single call but it works.

Reference from when I reported the same issue here. #1512

[LostBeard]

I am using SpawnDev.BlazorJS and below is an example of a working way to load an image's pixel data into a Mat. Essentially just using an Image to parse the image data, and then a canvas to get the pixel data to write to the Mat.

async Task<Mat> GetImageAsMat(string url, string? crossOrigin = "anonymous")
{
    using var image = await HTMLImageElement.CreateFromImageAsync(url, crossOrigin);
    using var canvas = new HTMLCanvasElement();
    using var context = canvas.Get2DContext();
    canvas.Width = image.Width;
    canvas.Height = image.Height;
    context.DrawImage(image, 0, 0);
    using var imageData = context.GetImageData(0, 0, image.Width, image.Height);
    using var uint8ClampedArray = imageData.Data;
    var rgbaBytes = uint8ClampedArray.ReadBytes();
    var mat = new Mat(new Size(image.Width, image.Height), MatType.CV_8UC4);
    Marshal.Copy(rgbaBytes, 0, mat.DataStart, rgbaBytes.Length);
    return mat;
}