SchemaDefinition with ImageDataViewType and Bitmaps

[josbri]

We are trying to use in memory images with the new ImageTypeAttribute, and it seems to work fine with const width and height. The problem is that we need the height/width to be assigned during runtime.

We've tried to create an SchemaDefinition, and assign the ColumnType to a new ImageDataViewType with the target height and width. Although the schema seems to be created properly, the CreatePredictionEngine method is returning the following error:

 System.NotSupportedException: Type 'System.Drawing.Bitmap' is not yet supported.
   en Microsoft.ML.Data.DataViewConstructionUtils.InputRowBase`1.CreateGetter(DataViewType colType, Column column, Delegate peek)
   en Microsoft.ML.Data.DataViewConstructionUtils.InputRowBase`1..ctor(IHostEnvironment env, DataViewSchema schema, InternalSchemaDefinition schemaDef, Delegate[] peeks, Func`2 predicate)
   en Microsoft.ML.Data.DataViewConstructionUtils.DataViewBase`1.DataViewCursorBase..ctor(IHostEnvironment env, DataViewBase`1 dataView, Func`2 predicate)
   en Microsoft.ML.Data.DataViewConstructionUtils.StreamingDataView`1.Cursor..ctor(IHostEnvironment env, StreamingDataView`1 dataView, Func`2 predicate)
   en Microsoft.ML.Data.DataViewConstructionUtils.StreamingDataView`1.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   en Microsoft.ML.Data.DataViewConstructionUtils.DataViewBase`1.GetRowCursorSet(IEnumerable`1 columnsNeeded, Int32 n, Random rand)
   en Microsoft.ML.Data.RowToRowMapperTransform.GetRowCursorSet(IEnumerable`1 columnsNeeded, Int32 n, Random rand)
   en Microsoft.ML.Data.RowToRowMapperTransform.GetRowCursorSet(IEnumerable`1 columnsNeeded, Int32 n, Random rand)
   en Microsoft.ML.Data.RowToRowMapperTransform.GetRowCursorSet(IEnumerable`1 columnsNeeded, Int32 n, Random rand)
   en Microsoft.ML.Data.DataViewUtils.TryCreateConsolidatingCursor(DataViewRowCursor& curs, IDataView view, IEnumerable`1 columnsNeeded, IHost host, Random rand)
   en Microsoft.ML.Data.TransformBase.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   en Microsoft.ML.Data.ColumnCursorExtensions.<GetColumnArrayDirect>d__4`1.MoveNext()
   en System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   en System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

I have seen some examples using in memory images, but none using non-const width and height values.

Source code / logs

internal class ImageData
 {
     [LoadColumn(0)]
     [ImageType(972, 1428)]
     public Bitmap Image;

     [LoadColumn(1)] public string Name;
 }

if the [ImageType(X,Y)] attribute is not added, we get the following error:

// When SchemaDefinition.Create(typeof(ImageData));

System.ArgumentOutOfRangeException: 'Could not determine an IDataView type and registered custom types for member Image
Nombre del parámetro: rawType'

With the attribute:

 _schemaDefinition = SchemaDefinition.Create(typeof(ImageData));
 _schemaDefinition[nameof(ImageData.Image)].ColumnType = new ImageDataViewType(config.ImageHeight, config.ImageWidth);

//Here is where we get the error:
_mlContext.Model.CreatePredictionEngine<ImageData, ImagePrediction>(_model, inputSchemaDefinition: _schemaDefinition);

Relevant Pipeline Code:

var imageResizing = _mlContext.Transforms.ResizeImages("ImageResized", config.ImageWidth, config.ImageHeight,
inputColumnName: nameof(ImageData.Image), resizing: ImageResizingEstimator.ResizingKind.Fill);

var pixelExtraction = _mlContext.Transforms.ExtractPixels(outputColumnName: config.InputLayerName,
inputColumnName: "ImageResized", interleavePixelColors: true);
var tfScoring = _mlContext.Model.LoadTensorFlowModel("path")
                          .ScoreTensorFlowModel(new[] { _outputLayerName }, new[] { config.InputLayerName });

return imageResizing.Append(pixelExtraction).Append(tfScoring).Fit(data);

[feiyun0112]

If you want to load/process the images in-memory, do not use LoadColumn attrbiute

public class ImageInputData
{
    [ImageType(ImageSettings.imageHeight, ImageSettings.imageWidth)]
    public Bitmap Image { get; set; }
}

please see sample: https://github.com/dotnet/machinelearning-samples/blob/main/samples/csharp/end-to-end-apps/ObjectDetection-Onnx/README.md

[josbri]

@feiyun0112 Unfortunately, the provided sample does use a const value for [ImageType(ImageSettings.imageHeight, ImageSettings.imageWidth)], which we are trying to avoid. Removing the LoadColumn attribute still prompts us with the same error (System.NotSupportedException: Type 'System.Drawing.Bitmap' is not yet supported) when trying to use the SchemaDefinition in the CreatePredictionEngine method.

public struct ImageSettings
{
    public const int imageHeight = 416;
    public const int imageWidth = 416;
}

public class ImageInputData
{
    [ImageType(ImageSettings.imageHeight, ImageSettings.imageWidth)]
    public Bitmap Image { get; set; }
}

This is in the provided sample, and we would like to not have to use a const value, since we want to be able to load different models.

[squirrelfeng]

I find that @josbri 's code actually works, only need to keep original [ImageType(X,Y)], then SchemaDefinition.Create will cover the original setting.