machinelearning icon indicating copy to clipboard operation
machinelearning copied to clipboard

PredictionPool will not let me have an MLImage member of an input or output type

Open tearlant opened this issue 1 year ago • 1 comments

System Information (please complete the following information):

  • Windows 10
  • ML.NET 3.0.0
  • .NET 7.0

Describe the bug There are a number of examples that use a PredictionEngine with an input type or output type that has a member of MLImage type, for example https://github.com/dotnet/machinelearning/issues/6886

However, my code is breaking when I call

var prePredictionEngine = mlContext.Model.CreatePredictionEngine<InputType, OutputType>(model);

    public class InputType
    {
        public string Label;

        [ColumnName("Image")]
        [ImageType(224, 224)]
        public MLImage Image { get; set; }
    }

This is the error that comes up image

Here's the console output

EXCEPTION
#########
System.NotImplementedException: Type 'Microsoft.ML.Data.MLImage' is not yet supported.
   at Microsoft.ML.Data.TypedCursorable`1.TypedRowBase.GenerateSetter(DataViewRow input, Int32 index, Column column, Delegate poke, Delegate peek)
   at Microsoft.ML.Data.TypedCursorable`1.TypedRowBase..ctor(TypedCursorable`1 parent, DataViewRow input, String channelMessage)
   at Microsoft.ML.Data.TypedCursorable`1.TypedRow..ctor(TypedCursorable`1 parent, DataViewRow input)
   at Microsoft.ML.Data.TypedCursorable`1.GetRow(DataViewRow input)
   at Microsoft.ML.PredictionEngineBase`2.PredictionEngineCore(IHostEnvironment env, InputRow`1 inputRow, IRowToRowMapper mapper, Boolean ignoreMissingColumns, SchemaDefinition outputSchemaDefinition, Action& disposer, IRowReadableAs`1& outputRow)
   at Microsoft.ML.PredictionEngineBase`2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition, Boolean ownsTransformer)
   at Microsoft.ML.PredictionEngine`2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition, Boolean ownsTransformer)
   at Microsoft.ML.PredictionEngineExtensions.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, IHostEnvironment env, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition, Boolean ownsTransformer)
   at Microsoft.ML.ModelOperationsCatalog.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
   at ImageClassification.ModelScorer.TFModelScorer.PredictDataUsingModelFromStructuredDirectory(IDataView testDataView) in C:\MLsamples\machinelearning-samples\samples\csharp\getting-started\DeepLearning_ImageClassification_TensorFlow\ImageClassification\ModelScorer\TFModelScorer.cs:line 181
   at ImageClassification.ModelScorer.TFModelScorer.ScoreUsingStructuredDirectory() in C:\MLsamples\machinelearning-samples\samples\csharp\getting-started\DeepLearning_ImageClassification_TensorFlow\ImageClassification\ModelScorer\TFModelScorer.cs:line 67
   at ImageClassification.Program.Main(String[] args) in C:\MLsamples\machinelearning-samples\samples\csharp\getting-started\DeepLearning_ImageClassification_TensorFlow\ImageClassification\Program.cs:line 32

I can make things work straight up by wrapping the data in a IDataView and calling .Transform(), but that defeats the point of having a PredictionEngine

tearlant avatar Jan 03 '24 03:01 tearlant

Update: I was able to create a workaround, by breaking things into two back to back PredictionEngines, the first of which is a pipeline with some data manipulation, then the first engine is just a ResizeImages and an ExtractImages transform, whose type is a VectorType instead of an ImageType.

But this feels like unnecessary gymnastics. I'm not sure what I'm doing wrong.

tearlant avatar Jan 03 '24 04:01 tearlant