machinelearning
machinelearning copied to clipboard
dotnet
Error while using Model.Save()
System Information (please complete the following information):
-
Checked on multiple systems (win 10, win 2019,2021 server)
-
ML.NET Version: ML.NET 1.7.0
-
.NET Core 2.2.3
Describe the bug When saving trained model of very large size, xx GB the exception is thrown with the inner exception "The length cannot be greater than the capacity. ParameterName valueCount at System.Text.StringBuilder.Append() at Microsoft.Ml.Data.ReadOnlyMemoryUtils.AppendSpan ...etc What I think is that the Save() uses StringBuilder.Append(char*,Int32) method and the int32 is a problem.
To Reproduce 1.Train extra large model 2.Try to save it with Model.Save()
@luisquintanilla It seems like this API is limiting the model size as intended, but there doesn't seem to be a workaround for users who want to save a large model. Is this functionality we need to add to ML.NET, or is there an alternative API I'm missing?
There is no alternative API that I'm aware of.
@ML-pixel do you experience these issues on newer versions of .NET? (i.e. .NET 6)
There is no alternative API that I'm aware of.
@ML-pixel do you experience these issues on newer versions of .NET? (i.e. .NET 6)
I will check it, but it's not a solution becasue it will make me upgrade production and I'm not sure if its possible right now.
@luisquintanilla I've tested it on .net 6 and the issue is still the same. @dakersnar Is there a chance it will be fixed anytime soon?
@ML-pixel thanks for confirming this is also the case on .NET 6. Do you know more or less how large your model is?
@dakersnar let's investigate and see what could be causing this. I added it to the ML.NET Future milestone for now. I suspect some of the new deep learning models might run into similar issues.
@ML-pixel thanks for confirming this is also the case on .NET 6. Do you know more or less how large your model is?
@dakersnar let's investigate and see what could be causing this. I added it to the ML.NET Future milestone for now. I suspect some of the new deep learning models might run into similar issues.
Well, for sure the zip file that was created just before crash was above 25 GB, the RAM consumption is around 400 GB.