Working most of the day, then suddenly this: import error for "tokenizers". Now won't start, 5mins ago working.

[ewebgh33]

Hi Was generating some XYZ plots to test various LORAs with Flux. It was all going fine most of the day, then after a few grids, it started generating black frames. Had not restarted or anything, it had been running a while, then boom, black images. Hmm weird. OK I will shut it down and open up forge-webui again, maybe it needs memory cleared or something.

Now this, on startup:

Traceback (most recent call last):
  File "C:\AI\Image\StableDiffusion\stable-diffusion-webui-forge\launch.py", line 54, in <module>
    main()
  File "C:\AI\Image\StableDiffusion\stable-diffusion-webui-forge\launch.py", line 50, in main
    start()
  File "C:\AI\Image\StableDiffusion\stable-diffusion-webui-forge\modules\launch_utils.py", line 546, in start
    import webui
  File "C:\AI\Image\StableDiffusion\stable-diffusion-webui-forge\webui.py", line 17, in <module>
    initialize_forge()
  File "C:\AI\Image\StableDiffusion\stable-diffusion-webui-forge\modules_forge\initialization.py", line 95, in initialize_forge
    modules_forge.patch_basic.patch_all_basics()
  File "C:\AI\Image\StableDiffusion\stable-diffusion-webui-forge\modules_forge\patch_basic.py", line 78, in patch_all_basics
    from transformers.dynamic_module_utils import logger
  File "C:\AI\Image\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\transformers\__init__.py", line 26, in <module>
    from . import dependency_versions_check
  File "C:\AI\Image\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\transformers\dependency_versions_check.py", line 57, in <module>
    require_version_core(deps[pkg])
  File "C:\AI\Image\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\transformers\utils\versions.py", line 117, in require_version_core
    return require_version(requirement, hint)
  File "C:\AI\Image\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\transformers\utils\versions.py", line 111, in require_version
    _compare_versions(op, got_ver, want_ver, requirement, pkg, hint)
  File "C:\AI\Image\StableDiffusion\stable-diffusion-webui\venv\lib\site-packages\transformers\utils\versions.py", line 44, in _compare_versions
    raise ImportError(
ImportError: tokenizers>=0.19,<0.20 is required for a normal functioning of this module, but found tokenizers==0.13.3.
Try: `pip install transformers -U` or `pip install -e '.[dev]'` if you're working with git main
Press any key to continue . . .

I had NOT updated today, so why did it work part of the day... then start giving me this all of a sudden?

So I rebooted PC... same error. PC, Win11, 4090 x 2. Can't get into the UI but terminal tells me Version: f2.0.1v1.10.1-previous-480-g3b9b2f65 Commit hash: 3b9b2f653ef9579e871e73f34b3c5037395cc351

Does this thing try to auto-update every time I run it? If so, how to turn that off? Can't be having 0-day problems all the time. Need something to run stable for a time then update when I choose. This makes no sense. I've not installed/updated ANYTHING today, nothing. So why was it working, then error?

Thanks

[ewebgh33]

Git pull to current version DOES NOTHING. Now on

Version: f2.0.1v1.10.1-previous-495-g4f64f6da
Commit hash: 4f64f6daa4582c8b5ddd5ccdb96a82fe86eaa91b

No positive effect, same error.

How can it suddenly be using a version of something that is wrong, when I

• did not update manually
• did not install anything else
• have been using the PC all day, so Windows has not done some random update to mess with things

Again, I literally have done nothing that would have caused a change. I was generating images, then got black frames, stopped webui, went to restart it, and then got this new error. How can it suddenly want a manual install of something when I did not initiate any changes myself?

[lllyasviel]

This log means you changed python environment. A typical mistake that can cause this is accidentally clicking wrong .bat file. You get this error when you were using run.bat and then mistakenly changed to webui.bat. Or, when you changed to administrator permission. Or when you changed something related to system python, or so.

[ewebgh33]

Thanks for the reply @lllyasviel . I run Forge from a shortcut on my desktop that links back to webui-user.bat. And have only ever, and always, use this shortcut. So I don't think I started the wrong bat... I have not done anything to the python env either. I was generating, it stopped generating, I closed terminal/session then started again from the shortcut. How would you recommend I fix this? As currently I can't start Forge.

[ewebgh33]

Interesting, it just started up. Ghosts in the machine. Maybe being turned off for 12hrs helped in some way. But a hard reboot/ power off seems to have done it. As an aside, while running Forge on GPU0, I did also try to run Auto1111 on GPU1 (2nd of my 2x 4090). Would that have messed it up? Each should be using it's own environment though.

[biktimirow]

I got the same issue ImportError: tokenizers>=0.19,<0.20 is required for a normal functioning of this module, but found tokenizers==0.15.2. Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git main

popped from nowhere, any ideas how to solve it?

[ewebgh33]

I suspect mine was caused by trying to run another python thing at the same time. Despite them being in different venvs / conda environments etc, and despite them using different GPUs, I don't know what else could have caused it. Mine was solved by a hard reboot (power off fully, not reset). OR I just think that solved it, and it was something else I'm not aware of doing or that happened automagically. If that helps at all.