swift-coreml-transformers icon indicating copy to clipboard operation
swift-coreml-transformers copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

Any plan for pretraining?

Open jbmaxwell opened this issue 6 years ago • 4 comments

I'm curious whether there's any plan to support pretrianing models from scratch?

jbmaxwell avatar Jul 27 '19 21:07 jbmaxwell

So, one way you could go about it, would be to use pytorch-transformers to pre-train or fine-tune your model, then use the script in model_generation as a starting point to convert to CoreML.

We do not have any plan to experiment with training on device, because realistically those models are way too large to be trained on anything other than a cutting-edge GPU 🙃

julien-c avatar Jul 29 '19 20:07 julien-c

Excellent, thanks for the reply! I wondered about using gpt2.py as a starting point... Of course, I wouldn't expect to train one of these beasts on the device! That would be madness! Maybe one day... haha... Just curious; would a model generated in this way be iOS 13+ only, or are the basic layer/objects compatible with iOS 12?

jbmaxwell avatar Jul 29 '19 20:07 jbmaxwell

Check out this tweet: https://twitter.com/julien_c/status/1154894146328563715

Short answer: iOS 13+ only

julien-c avatar Jul 29 '19 20:07 julien-c

Ah, too bad, but not surprising. The new hotness is the new hotness for a reason! ;-)

jbmaxwell avatar Jul 29 '19 20:07 jbmaxwell