Sylvain Gugger

Mmmm, before merging this PR, there is something wrong going on with the tests: only one of the tests job is actually run (no tests_flax/tests_tf etc...) Will investigate later today...

Thanks @andyehrenberg ! @sanchit-gandhi Can you have one final look?

cc @gante and @sanchit-gandhi

There is no problem with the model requiring torchvision to be installed. We have many models with specific dependencies, some of which you ported yourself ;-). Just protect the import...

In this case, remove the code trying to load the custom kernels in the modeling file and we can add it back in the PR that will deal with custom...

> I think decode(encode(text)) == text should be true by default This is untrue for pretty much all tokenizers, since tokenization is a destructive operation. At the very least you...

cc @gante

cc @alaradirik and @amyeroberts

I don't think it's possible to get the sentencepiece model from the `tokenizer.json` file but maybe @Narsil knows a way.

Hi there, what you are asking for is not supported. Note that Transformers is primarily a library of models. You can adapt the data preprocessing part of any of our...