coremltools
coremltools copied to clipboard
apple
Slow model loading
Hey, I have been using coremltools for some time and model loading has been very fast, usually within seconds. I am using an ~150Mbs size mlpackage model with coremltool version 5.2.0. Recently, I upgraded to 6.2 and after this, the coreml model loading increased to 150secs. Could anyone who faced a similar issue at any time during development help me in this regard. my code is:
import coremltools as ct
model = ct.models.MLModel(model_path)
Thanks in advance!
system is mac m1 air, 8Gb mem
I've never heard of something like this before. How fast was your model loading with coremltools 5.2? How long is it loading with 6.2? Are these numbers consistent?
Hey, So the issue was with numpy version. With coremltools==5.2 I have numpy==1.23.1, and 150Mb model loading was in 1 2 seconds. Then I updated coremltools to the latest 6.2 and the same model loading started taking around 90 seconds.
I'm loading CLIP image and text models and they are also extremely slow to load on ANE (CPU loads them instantly). 335M and 242M. On an iPhone 14 Pro Max it takes 97s for the image model then just 7s for the text model following it. It is so bad I may just use the CPU models even though performance suffers.
By numpy version? Or by coremltools version?
no, numpy version, also the protobuf version.
I'm loading CLIP image and text models and they are also extremely slow to load on ANE (CPU loads them instantly). 335M and 242M. On an iPhone 14 Pro Max it takes 97s for the image model then just 7s for the text model following it. It is so bad I may just use the CPU models even though performance suffers.
not sure about ane, but could you try playing little with coremltools, numpy, protobuf versions. On m1 mac it takes roughly 10-15 secs to load clip image model (335M)
Yes, that is how long it takes on an M1 Mac. Much longer on the phone. I've added a thing where it uses CPU until it finishes loading it into ANE in the background.
Just wanted to call out that loading these models is basically unusuable on the iPhone as it often gets killed in the background and restarts almost every time you leave the application.
I see the same thing as @spullara with CLIP image and text models. iPhone 14 Pro running iOS 17.4.
@bcherry funny seeing you in the same thread! I ended up using the CPU model until the ANE model finishes loading to avoid making the user wait. https://twitter.com/getrememberwhen