text-generation-webui

text-generation-webui copied to clipboard

Describe the bug

I can't get the api to work. Either it says ConnectionRefused or when you change the port to 7860 it tells me some strange html errors. I already deleted everything and reinstalled it but still the same issue.

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

Using latest branch, run api-example-stream.py and you will get this issue.

Screenshot

No response

Logs

Start Webui:
cybertimon@server:~/Repositorys/text-generation-webui$ python3 server.py --model vicuna-1.1-13b-4bit-128g --gpu-memory 12 --wbits 4 --groupsize 128 --listen-host 0.0.0.0 --listen --xformers --extension api --listen-port 21129
Gradio HTTP request redirected to localhost :)
Loading vicuna-1.1-13b-4bit-128g...
Found the following quantized model: models/vicuna-1.1-13b-4bit-128g/vicuna-13B-1.1-GPTQ-4bit-128g.no-act-order.pt
Loading model ...
Done.
Using the following device map for the quantized model: {'': 0}
Replaced attention with xformers_attention
Loaded the model in 4.01 seconds.
Loading the extension "api"... Ok.
Starting KoboldAI compatible api at http://0.0.0.0:5000/api
Running on local URL:  http://0.0.0.0:21129

To create a public link, set `share=True` in `launch()`.

Run api-example-stream.py without changes: ################################################! <-

ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 7860)

After changing API Port in api-example-stream.py to 5000: #################################! <-

Traceback (most recent call last):
  File "/home/cybertimon/miniconda3/lib/python3.10/site-packages/websockets/legacy/client.py", line 139, in read_http_response
    status_code, reason, headers = await read_response(self.reader)
  File "/home/cybertimon/miniconda3/lib/python3.10/site-packages/websockets/legacy/http.py", line 130, in read_response
    raise ValueError(f"unsupported HTTP version: {d(version)}")
ValueError: unsupported HTTP version: HTTP/1.0

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cybertimon/Repositorys/text-generation-webui/api-example-stream.py", line 92, in <module>
    asyncio.run(get_result())
  File "/home/cybertimon/miniconda3/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/cybertimon/miniconda3/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/cybertimon/Repositorys/text-generation-webui/api-example-stream.py", line 85, in get_result
    async for response in run(prompt):
  File "/home/cybertimon/Repositorys/text-generation-webui/api-example-stream.py", line 53, in run
    async with websockets.connect(f"ws://{server}:5000/queue/join") as websocket:
  File "/home/cybertimon/miniconda3/lib/python3.10/site-packages/websockets/legacy/client.py", line 637, in __aenter__
    return await self
  File "/home/cybertimon/miniconda3/lib/python3.10/site-packages/websockets/legacy/client.py", line 655, in __await_impl_timeout__
    return await self.__await_impl__()
  File "/home/cybertimon/miniconda3/lib/python3.10/site-packages/websockets/legacy/client.py", line 662, in __await_impl__
    await protocol.handshake(
  File "/home/cybertimon/miniconda3/lib/python3.10/site-packages/websockets/legacy/client.py", line 323, in handshake
    status_code, response_headers = await self.read_http_response()
  File "/home/cybertimon/miniconda3/lib/python3.10/site-packages/websockets/legacy/client.py", line 145, in read_http_response
    raise InvalidMessage("did not receive a valid HTTP response") from exc
websockets.exceptions.InvalidMessage: did not receive a valid HTTP response

System Info

Ubuntu 20.04
2x RTX 3060 12gb
I5 13600kf
64gb ddr4

try the kobold api

localhost:5000/api/v1/generate

How does this work with streaming?

doesnt as far as i know

API not working for me either. i get the following message {"detail":"Method Not Allowed"} at http://127.0.0.1:7860/run/textgen. the inbuilt api widget within the webui fails, so do external calls.

try this： https://github.com/oobabooga/text-generation-webui/discussions/1137

It worked, thank you ser!

Closing as this is fixed with the new api extension