Tokenizer version is in conflict with transformers version

[GanGral]

          I followed the same instructions at: https://microsoft.github.io/AI-For-Beginners/etc/how-to-run

This command took hours to execute and then failed after having successfully completed other steps: conda env create --name ai4beg --file .devcontainer/environment.yml

Error is as follows:

The conflict is caused by: The user requested tokenizers==0.10.3 transformers 4.30.0 depends on tokenizers!=0.11.3, <0.14 and >=0.11.1

To fix this you could try to:

1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

Pip subprocess error: ERROR: Cannot install -r C:\Users\lufonzi\Python\AI-For-Beginners.devcontainer\requirements.txt (line 22) and tokenizers==0.10.3 because these package versions have conflicting dependencies. ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

failed

CondaEnvException: Pip failed

Originally posted by @lufonzi in https://github.com/microsoft/AI-For-Beginners/issues/201#issuecomment-1666562597

[GanGral]

Fixing tokenizer version in .devcontainer/requirements.txt should fix the issue

[omar908]

@GanGral I reproduced the issue, I will commit the change once I confirm it fixed the issue and will create the Pull Request on this one, thank you!

[l3ka]

@omar908 This issues is still a thing. And while trying to for example, pip install -r ./.devcontainer/requirements.txt following message will pop up

The conflict is caused by:
    The user requested tokenizers==0.10.3
    transformers 4.30.0 depends on tokenizers!=0.11.3, <0.14 and >=0.11.1

Something like this in requirements.txt file, would resolve the issue tokenizers>=0.11.1,!=0.11.3,<0.14 or like in the suggested PR tokenizers==0.13.3. Can you check on the above linked PR?

[omar908]

Hey @l3ka , the PR above is the one I created a while back, but seems it is still awaiting approval from someone of this project. Won't allow me to merge it until then, not much I can do on my end, sadly.

[l3ka]

Hey @omar908, thanks for coming back this quickly. Yes, that make sense a lot, so unfortunately we will have to wait for someone to approve that PR.

I am planning to check out this course, in following week or two, so if I encounter any other issue (already did), I will try to summarise all of them, in one Issue post.