swift-coreml-transformers icon indicating copy to clipboard operation
swift-coreml-transformers copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

CLIPTokenizer

Open MatthewWaller opened this issue 2 years ago • 3 comments

Hi y'all, I was looking to convert the CLIP Tokenizer found here.

I noticed it seems a lot like the GPT2Tokenizer.swift.

Are there perchance simple adjustments I could make so that it would work for a CLIP model?

Thanks!

MatthewWaller avatar Sep 14 '22 17:09 MatthewWaller

Any update on this? I too am interested.

Lukas1h avatar Oct 09 '22 18:10 Lukas1h

Also interested!

taydr avatar Jan 31 '23 19:01 taydr

I have a PR of the tokenizer I used if y'all need it. Take a look here! https://github.com/huggingface/swift-coreml-transformers/pull/30

MatthewWaller avatar Jan 31 '23 21:01 MatthewWaller