Peter

It is now merged and released in v0.1.19, please give it a try. Let me know if there are any problems, feature requests, or any inconvenience when using that.

You seems to be using the old version, please update to the latest one. And it should be used like this: ```julia using Transformers using Transformers.Basic using Transformers.HuggingFace textenc =...

It really depends on the scenario. Would need to expand on that (e.g. what do you have and what do you want to achieve).

Hi! First of all, thanks for the contribution. It's glad to see something/someone new coming in. However, I'm a little busy these days, so I won't be able to do...

Don't worry about the time and docs, do it with your own pace. Here are some stuffs I think that it can improve. 1. some function/module should be imported into...

I was planning about adding these stuffs myself after I fix the Zygote bugs, but I got too busy to do that. Here is the only file I wrote that...

looks like something crash with CuArrays, but I can't reproduce the error. What's your version?

I’m using Julia 1.4 with CuArrays 1.7.3. Maybe try to update CuArrays to 1.7.3 (btw, Transformers is now 0.1.3)

The compat bound on CuArrays 1.6 because I was using and testing with Julia 1.3

The new tokenizer api (using TextEncodeBase) is basically finished and included in the 0.1.16 release, though the gpt part is ignored for now. For the next step, I will be...