machinelearning
machinelearning copied to clipboard
SchemaDefinition with ImageDataViewType and Bitmaps
We are trying to use in memory images with the new ImageTypeAttribute, and it seems to work fine with const width and height. The problem is that we need the height/width to be assigned during runtime.
We've tried to create an SchemaDefinition, and assign the ColumnType to a new ImageDataViewType with the target height and width. Although the schema seems to be created properly, the CreatePredictionEngine method is returning the following error:
System.NotSupportedException: Type 'System.Drawing.Bitmap' is not yet supported.
en Microsoft.ML.Data.DataViewConstructionUtils.InputRowBase`1.CreateGetter(DataViewType colType, Column column, Delegate peek)
en Microsoft.ML.Data.DataViewConstructionUtils.InputRowBase`1..ctor(IHostEnvironment env, DataViewSchema schema, InternalSchemaDefinition schemaDef, Delegate[] peeks, Func`2 predicate)
en Microsoft.ML.Data.DataViewConstructionUtils.DataViewBase`1.DataViewCursorBase..ctor(IHostEnvironment env, DataViewBase`1 dataView, Func`2 predicate)
en Microsoft.ML.Data.DataViewConstructionUtils.StreamingDataView`1.Cursor..ctor(IHostEnvironment env, StreamingDataView`1 dataView, Func`2 predicate)
en Microsoft.ML.Data.DataViewConstructionUtils.StreamingDataView`1.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
en Microsoft.ML.Data.DataViewConstructionUtils.DataViewBase`1.GetRowCursorSet(IEnumerable`1 columnsNeeded, Int32 n, Random rand)
en Microsoft.ML.Data.RowToRowMapperTransform.GetRowCursorSet(IEnumerable`1 columnsNeeded, Int32 n, Random rand)
en Microsoft.ML.Data.RowToRowMapperTransform.GetRowCursorSet(IEnumerable`1 columnsNeeded, Int32 n, Random rand)
en Microsoft.ML.Data.RowToRowMapperTransform.GetRowCursorSet(IEnumerable`1 columnsNeeded, Int32 n, Random rand)
en Microsoft.ML.Data.DataViewUtils.TryCreateConsolidatingCursor(DataViewRowCursor& curs, IDataView view, IEnumerable`1 columnsNeeded, IHost host, Random rand)
en Microsoft.ML.Data.TransformBase.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
en Microsoft.ML.Data.ColumnCursorExtensions.<GetColumnArrayDirect>d__4`1.MoveNext()
en System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
en System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
I have seen some examples using in memory images, but none using non-const width and height values.
Source code / logs
internal class ImageData
{
[LoadColumn(0)]
[ImageType(972, 1428)]
public Bitmap Image;
[LoadColumn(1)] public string Name;
}
if the [ImageType(X,Y)] attribute is not added, we get the following error:
// When SchemaDefinition.Create(typeof(ImageData));
System.ArgumentOutOfRangeException: 'Could not determine an IDataView type and registered custom types for member Image
Nombre del parámetro: rawType'
With the attribute:
_schemaDefinition = SchemaDefinition.Create(typeof(ImageData));
_schemaDefinition[nameof(ImageData.Image)].ColumnType = new ImageDataViewType(config.ImageHeight, config.ImageWidth);
//Here is where we get the error:
_mlContext.Model.CreatePredictionEngine<ImageData, ImagePrediction>(_model, inputSchemaDefinition: _schemaDefinition);
Relevant Pipeline Code:
var imageResizing = _mlContext.Transforms.ResizeImages("ImageResized", config.ImageWidth, config.ImageHeight,
inputColumnName: nameof(ImageData.Image), resizing: ImageResizingEstimator.ResizingKind.Fill);
var pixelExtraction = _mlContext.Transforms.ExtractPixels(outputColumnName: config.InputLayerName,
inputColumnName: "ImageResized", interleavePixelColors: true);
var tfScoring = _mlContext.Model.LoadTensorFlowModel("path")
.ScoreTensorFlowModel(new[] { _outputLayerName }, new[] { config.InputLayerName });
return imageResizing.Append(pixelExtraction).Append(tfScoring).Fit(data);
If you want to load/process the images in-memory, do not use LoadColumn attrbiute
public class ImageInputData
{
[ImageType(ImageSettings.imageHeight, ImageSettings.imageWidth)]
public Bitmap Image { get; set; }
}
please see sample: https://github.com/dotnet/machinelearning-samples/blob/main/samples/csharp/end-to-end-apps/ObjectDetection-Onnx/README.md
@feiyun0112 Unfortunately, the provided sample does use a const value for [ImageType(ImageSettings.imageHeight, ImageSettings.imageWidth)], which we are trying to avoid. Removing the LoadColumn attribute still prompts us with the same error (System.NotSupportedException: Type 'System.Drawing.Bitmap' is not yet supported) when trying to use the SchemaDefinition in the CreatePredictionEngine method.
public struct ImageSettings
{
public const int imageHeight = 416;
public const int imageWidth = 416;
}
public class ImageInputData
{
[ImageType(ImageSettings.imageHeight, ImageSettings.imageWidth)]
public Bitmap Image { get; set; }
}
This is in the provided sample, and we would like to not have to use a const value, since we want to be able to load different models.
I find that @josbri 's code actually works, only need to keep original [ImageType(X,Y)], then SchemaDefinition.Create will cover the original setting.