users on windows have an issue with chat generation on alpaca/vicuna/gpt4 models

[Tom-Neverwinter]

Describe the bug

AMD/ windows users are affected since about 4 days ago.

This is a recurring theme.

they will load the model and then it outputs gibberish.

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

any amd/windows system

Screenshot

No response

Logs

same as all windows users. cuda/nvidia related

System Info

https://pastebin.com/wRVCpcep full parts list

[Tom-Neverwinter]

based on a report from five days ago this relates to users on windows. issue still uncertain.

windows, nvidia.

torch 2.0.0 and cuda version issue?

https://github.com/oobabooga/text-generation-webui/issues/1040 [similar setup] user states works in 8bit but not 4bit

https://huggingface.co/blog/hf-bitsandbytes-integration

https://github.com/oobabooga/text-generation-webui/issues/1189

https://github.com/oobabooga/text-generation-webui/issues/1284

is it related to cuda? test cuda 11.2 failed cuda_11.2.0_460.89_win10 test cuda 11.3 test cuda 11.4 test cuda 11.5 test cuda 11.6 test cuda 11.7 test cuda 11.8 failed

test windows drivers tesla m40 has limited drivers though :/ nvidia gtx 1080ti drivers?

multi driver setup issue?

https://github.com/oobabooga/text-generation-webui/issues/931 Have you tried removing the non-cuda .pt file out of the directory and only having the cuda version? That solved the gibberish for me

tokenizer issue??: https://huggingface.co/Neko-Institute-of-Science

[floppaselfbot]

im using wsl so im not running it on windows by itself, i assume thats how it works

[floppaselfbot]

but yeah im still experiencing this issue and i have not gotten it to work as of yet

[Tom-Neverwinter]

https://github.com/ClayShoaf/oobabooga-one-click-bandaid

I have not tested this but this is claimed to fix the issue

moved to newer solution: https://github.com/xNul/one-click-installers

make sure one version of cuda is installed like 11.8 [tesla m40 cards] and cuda 12.1 for newer cards

[Phosay]

I am not quite sure if this is related but the OPT-derived model like opt-6.7b-Erebus with flexgen and compress weight arguments seems to either give answers like "?????" or types something that gets quickly deleted when next part gets written, console claims all tokens were generated but I can only see last few words or signs, in the latter case the argument verbose shows that model gave no answer at all even with tokens generated; I am on Windows 10 with RTX 2060 6 GB VRAM (used both commit 91745f6 as well as older installer one you posted here)

Edit: So, I manually installed various commits to test where it breaks: 143e886 (from 21.04), 7ff6458 (from 22.04), 2f4f124 (from 24.04) and those seem to be working correctly so far but seems that commit da81260 (from 25.04) is giving answers only with 8, 16 or 32 tokens max so I feel something went wrong around that time; I might have figured out something, commit 1a0c12c (from 25.04) the one that refractored text-generation.py seems to be affecting the tokens generated negatively, when I reverted it to the one from commit 2f4f124 it works properly (it also works with commit 91745f6).

[Tom-Neverwinter]

https://github.com/xNul/one-click-installers

make sure one version of cuda is installed like 11.8 [tesla m40 cards] and cuda 12.1 for newer cards

[madwurmz]

for windows nvidia , gibberish and cuda missing error, can be fixed with these instructions:

install the newest ogaabogaa 1-click-installer then do this:

1. open cmd_windows.bat
2. pip uninstall quant-cuda
3. cd text-generation-webui\repositories
4. rm -f -d -r GPTQ-for-LLaMa
5. git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa -b cuda
6. cd GPTQ-for-LLaMa
7. python setup_cuda.py install
8. close the cmd and run the start_windows.bat like normal

[Phosay]

I tried it this way on the ee68ec9 commit, although the rm -f -d -r commands don't work on cmd on windows, I just deleted that folder on my own and then git cloned the repository, it seems to be working properly but installer itself had quite a lot of various errors.

[teknium1]

Since its fixed by reverting hf transformers, I dont think its cuda related?

[madwurmz]

rm -f -d -r commands don't work on cmd

the cmd should be still from step 1, that relies on Miniconda, it was surely working for me

[madwurmz]

Since its fixed by reverting hf transformers, I dont think its cuda related?

I'm sorry I didnt mean to be unrelated. I was just too happy to share some solution I found for my gibberish problem, so I searched for all obagobaa related gibberish topics.

[github-actions[bot]]

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.