Transformers.jl icon indicating copy to clipboard operation
Transformers.jl copied to clipboard

DistilBertModel support

Open AbrJA opened this issue 1 year ago • 1 comments

Hi there,

First of all, thanks for the wonderful package you have done.

I'm trying to load a HuggingFace Model but I got these warnings and an error:

using Transformers
using Transformers.TextEncoders
using Transformers.HuggingFace

textenc, model = hgf"sentence-transformers/distiluse-base-multilingual-cased-v1"

┌ Warning: startsym <s> not in vocabulary, this might cause problem.
└ @ Transformers.TextEncoders [/home/ajaimes/.julia/packages/Transformers/lD5nW/src/textencoders/bert_textencoder.jl:77](https://file+.vscode-resource.vscode-cdn.net/home/ajaimes/.julia/packages/Transformers/lD5nW/src/textencoders/bert_textencoder.jl:77)

┌ Warning: endsym </s> not in vocabulary, this might cause problem.
└ @ Transformers.TextEncoders [/home/ajaimes/.julia/packages/Transformers/lD5nW/src/textencoders/bert_textencoder.jl:78](https://file+.vscode-resource.vscode-cdn.net/home/ajaimes/.julia/packages/Transformers/lD5nW/src/textencoders/bert_textencoder.jl:78)

Unknown model type: distilbert

Do you have in mind adding support for this type of models (distilbert)? Or is there some way to achieve this?

Thank you in advance!

AbrJA avatar Jan 03 '24 22:01 AbrJA