Connection errored out

[Saoma1]

Describe the bug

So I have the web-ui finally running, now I encounter a Connection errored out every time I try to load a model.

I have tried it with the gpt4-x-alpaca and the vicuna

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

Load model in the web-ui

Screenshot

Logs

python3 server.py --auto-device --chat --wbits 4 --groupsize 128
INFO:Gradio HTTP request redirected to localhost :)
INFO:Loading the extension "gallery"...
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
INFO:Loading gpt4-x-alpaca-13b-native-4bit-128g...
INFO:Found the following quantized model: models/gpt4-x-alpaca-13b-native-4bit-128g/gpt-x-alpaca-13b-native-4bit-128g-cuda.pt
[1]    104215 killed     python3 server.py --auto-device --chat --wbits 4 --groupsize 128

System Info

I have a IdeaPad 5 15ARE05 laptop

Ubuntu 20.04.6 LTS

AMD Ryzen 5 4500U
16gb Ram

[LaaZa]

I don't think that laptop even has a dedicated GPU and it's AMD. And eitherway it would be very unlikely you could load so large model on anything but the best gaming laptops, barely. You can try a GGML model that runs on the cpu, but it will be slow and you don't have that much ram either. Not the best computer to run LLMs on. Try a smaller 7B model for example and use the GGML models.

[derek-upham]

This seems similar to https://github.com/vladmandic/automatic/issues/205, which is supposedly the same as https://github.com/oobabooga/text-generation-webui/issues/819, which is supposedly fixed by https://github.com/oobabooga/text-generation-webui/pull/1089.

Do you get the same error if you simply click on the "refresh" button next to the Models drop-down list?

What is your browser?

[LaaZa]

This seems similar to https://github.com/vladmandic/automatic/issues/205, which is supposedly the same as https://github.com/oobabooga/text-generation-webui/issues/819, which is supposedly fixed by https://github.com/oobabooga/text-generation-webui/pull/1089.

Do you get the same error if you simply click on the "refresh" button next to the Models drop-down list?

What is your browser?

No it is not. The process is killed by the os because it tried to use way too much more memory than what was available.

[derek-upham]

Yeah, the above error is actually printing something to the console log.

My own debugging quest continues.

[Saoma1]

@LaaZa thank you for the notice. The laptop dose indeed not have a dedicated gpu. I was of the assumption that this repo in combination with a 4bit model would allow me to run it locally. Only recently I got interested in running LLMs locally, so I will do some more reading and check out the models you talked about.