What does this PR do?

This PR adds test tokenization file for Splinter. It inherits from BERT Tokenizer.

Contributes fixes to issue https://github.com/huggingface/transformers/issues/16627

Before submitting

• [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• [x] Did you read the contributor guideline, Pull Request section?
• [x] Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• [ ] Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
• [ ] Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

@LysandreJik @SaulLu

The documentation is not available anymore as the PR was closed or merged.

Hi @farahdian , thank you very much for your contribution. I see that several tests have failed, is this still a work in progress?

Hi @farahdian , thank you very much for your contribution. I see that several tests have failed, is this still a work in progress?

Yup a work in progress, but will appreciate some guidance and will be inspecting the failed tests. Sorry for any confusion!

Ok top! I'd be happy to give you a hand.

I think in your case it would be great if the title of the PR started with [WIP] and a first failing test that you can fix is style of the files by running the make fixup command locally (cf the contributor guideline).

For the rest of the tests that fail, could you tell me more about what is obscure for you?

Many thanks. I've tried to run make fixup but this error keeps coming up:

make : The term 'make' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
+ make fixup
+ ~~~~
    + CategoryInfo          : ObjectNotFound: (make:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

I've tried using python -m alongside but then it returns this:

usage: __main__.py [-h] {project,get} ...
__main__.py: error: invalid choice: 'fixup' (choose from 'project', 'get')

Think this may be related to why I've been having some issues running tests locally... appreciate you having a look!

From your error message, what I understand is that you don't have the make command installed on your computer.

(source)

On Unix-like operating systems, make is a utility for building and maintaining groups of programs (and other types of files) from source code.

Depending on your OS, you'll probably have an alternative to install it. For example on Windows you can use WSL.

Hi @farahdian ,

Just a quick message to see how you're doing with adding the tests on your end. :relaxed:

Hi @farahdian ,

Just a quick message to see how you're doing with adding the tests on your end. ☺️

Thanks for checking up on me!

I'm struggling a bit with this and I'm not sure how to proceed... I've been trying to use WSL and it seems like I'm coming close but this error appears.

make: *** No rule to make target 'fixup'. Stop.

Hi @farahdian ,

Thanks for the update. What is your working directory when you run the make fixup command?

Hi @farahdian ,

Thanks for the update. What is your working directory when you run the make fixup command?

transformers/tests/splinter

I see, you need to run it from the root repository transformers/ where the Makefile lives :blush:

I see, you need to run it from the root repository transformers/ where the Makefile lives 😊

Unfortunately, when I run '''make fixup''' from the root repo this error then comes up:

make: python: Command not found
No library .py files were modified
python utils/custom_init_isort.py
make: python: Command not found
make: *** [Makefile:56: extra_style_checks] Error 127

This error suggests that python isn't installed. I guess that you'll get the same error if you run python --version (which isn't specific to transformers)

Hi @farahdian,

How things are going for you? :slightly_smiling_face:

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

Sorry for the delay, think I reached beyond my capabilities with this one. Hope this can be passed on to another contributer

Thank you for keeping us informed