CLIPTokenizer behaves inconsistently depending on whether ftfy is installed or not

[kjsman]

System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

• transformers version: 4.22.1
• Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
• Python version: 3.7.14
• Huggingface_hub version: 0.9.1
• PyTorch version (GPU?): 1.12.1+cu113 (False)
• Tensorflow version (GPU?): 2.8.2 (False)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

Who can help?

@patil-suraj

Information

• [ ] The official example scripts
• [ ] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

1. Run following code without ftfy installed.

from transformers import CLIPTokenizer
tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14")
tokenizer("résumé") # {'input_ids': [49406, 15077, 49407], 'attention_mask': [1, 1, 1]}

2. Run following code with ftfy installed.

from transformers import CLIPTokenizer
tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14")
tokenizer("résumé") # {'input_ids': [49406, 29106, 7054, 4166, 49407], 'attention_mask': [1, 1, 1, 1, 1]}

Expected behavior

They should work consistently.

[kjsman]

This happens because BasicTokenizer, which is used as fallback text fix function, strips accents if do_lower_case=True.

We may fix this by explicitly set strip_accents to False; ViT/L-14 tokenizer includes vocabs with accents, so I think stripping accents should not be done.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.