Error: Connection Refused by SGLANG Backend

[deadpipe]

Hello

I am trying to launch the Llava-1.6-34b model on WSL2 (ubuntu 22.04)

I have launched 4 instances of command prompt and entered the following commands:

1. python -m llava.serve.controller --host 0.0.0.0 --port 10000

2. python -m llava.serve.gradio_web_server --controller http://localhost:10000 --model-list-mode reload

3. CUDA_VISIBLE_DEVICES=0,1 python3 -m sglang.launch_server --model-path "/mnt/s/Models/llava-v1.6-34b" --tokenizer-path liuhaotian/llava-v1.6-34b-tokenizer --port 30000 --tp 2

4. python -m llava.serve.sglang_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --sgl-endpoint http://127.0.0.1:30000

after I enter the 4th command, SGL Backend refuses the connection. I get this error :

(llava) badapadda@DESKTOP-UHHHGJ4:~/llava/LLaVA$ python -m llava.serve.sglang_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --sgl-endpoint http://localhost:30000 2024-03-06 02:16:59 | INFO | model_worker | args: Namespace(host='0.0.0.0', port=40000, worker_address='http://localhost:40000', controller_address='http://localhost:10000', model_name=None, sgl_endpoint='http://localhost:30000', limit_model_concurrency=5, stream_interval=1, no_register=False) 2024-03-06 02:16:59 | ERROR | stderr | Traceback (most recent call last): 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/urllib/request.py", line 1348, in do_open 2024-03-06 02:16:59 | ERROR | stderr | h.request(req.get_method(), req.selector, req.data, headers, 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/http/client.py", line 1283, in request 2024-03-06 02:16:59 | ERROR | stderr | self._send_request(method, url, body, headers, encode_chunked) 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/http/client.py", line 1329, in _send_request 2024-03-06 02:16:59 | ERROR | stderr | self.endheaders(body, encode_chunked=encode_chunked) 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/http/client.py", line 1278, in endheaders 2024-03-06 02:16:59 | ERROR | stderr | self._send_output(message_body, encode_chunked=encode_chunked) 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/http/client.py", line 1038, in _send_output 2024-03-06 02:16:59 | ERROR | stderr | self.send(msg) 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/http/client.py", line 976, in send 2024-03-06 02:16:59 | ERROR | stderr | self.connect() 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/http/client.py", line 942, in connect 2024-03-06 02:16:59 | ERROR | stderr | self.sock = self._create_connection( 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/socket.py", line 845, in create_connection 2024-03-06 02:16:59 | ERROR | stderr | raise err 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/socket.py", line 833, in create_connection 2024-03-06 02:16:59 | ERROR | stderr | sock.connect(sa) 2024-03-06 02:16:59 | ERROR | stderr | ConnectionRefusedError: [Errno 111] Connection refused 2024-03-06 02:16:59 | ERROR | stderr | 2024-03-06 02:16:59 | ERROR | stderr | During handling of the above exception, another exception occurred: 2024-03-06 02:16:59 | ERROR | stderr | 2024-03-06 02:16:59 | ERROR | stderr | Traceback (most recent call last): 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-03-06 02:16:59 | ERROR | stderr | return _run_code(code, main_globals, None, 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/runpy.py", line 86, in _run_code 2024-03-06 02:16:59 | ERROR | stderr | exec(code, run_globals) 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/llava/LLaVA/llava/serve/sglang_worker.py", line 238, in 2024-03-06 02:16:59 | ERROR | stderr | worker = ModelWorker(args.controller_address, 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/llava/LLaVA/llava/serve/sglang_worker.py", line 62, in init 2024-03-06 02:16:59 | ERROR | stderr | backend = RuntimeEndpoint(sgl_endpoint) 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/site-packages/sglang/backend/runtime_endpoint.py", line 22, in init 2024-03-06 02:16:59 | ERROR | stderr | res = http_request( 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/site-packages/sglang/utils.py", line 110, in http_request 2024-03-06 02:16:59 | ERROR | stderr | resp = urllib.request.urlopen(req, data=data) 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/urllib/request.py", line 216, in urlopen 2024-03-06 02:16:59 | ERROR | stderr | return opener.open(url, data, timeout) 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/urllib/request.py", line 519, in open 2024-03-06 02:16:59 | ERROR | stderr | response = self._open(req, data) 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/urllib/request.py", line 536, in _open 2024-03-06 02:16:59 | ERROR | stderr | result = self._call_chain(self.handle_open, protocol, protocol + 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/urllib/request.py", line 496, in _call_chain 2024-03-06 02:16:59 | ERROR | stderr | result = func(*args) 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/urllib/request.py", line 1377, in http_open 2024-03-06 02:16:59 | ERROR | stderr | return self.do_open(http.client.HTTPConnection, req) 2024-03-06 02:16:59 | ERROR | stderr | File "/home/badapadda/miniconda3/envs/llava/lib/python3.10/urllib/request.py", line 1351, in do_open 2024-03-06 02:16:59 | ERROR | stderr | raise URLError(err) 2024-03-06 02:16:59 | ERROR | stderr | urllib.error.URLError: <urlopen error [Errno 111] Connection refused>

See Screenshot below :

I have also dual-booted my computer with Ubuntu 22.04, and tried the same steps and I have got the exact same error.

Any help is highly appreciated

[Z500-RAY]

same problem

[deadpipe]

same problem

@Z500-RAY have you found a solution?

[github-actions[bot]]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.

[deadpipe]

Has someone found a solution?

[SusanLiu0709]

Same issue here.

[Edenzzzz]

Probably because you need to wait for the server to fully launch before sending requests.