FastChat
FastChat copied to clipboard
lm-sys
ERROR: [Errno 99] error while attempting to bind on address ('::1', 21001, 0, 0): cannot assign requested address
ERROR: [Errno 99] error while attempting to bind on address ('::1', 21001, 0, 0): cannot assign requested address
@Ted8000 thank u , currently fschat start an openai servce is very cubersome, I need start controller, than start worker, than start runner, is there a way just run a single script and do them all??
Just create a script that starts off of them!
@Ted8000 thank u , currently fschat start an openai servce is very cubersome, I need start controller, than start worker, than start runner, is there a way just run a single script and do them all??
Just create a shell script which begins all of them for you!
To me, it makes a lot of sense this separation. You can keep the controller running, add and remove models in other machines and in more gpus, and keep the UI running isolated from that. It's the best way of implementing this, to be honest.
@surak Yes, from this aspect of view, it's scalable. But now, I just have one single model to serve, and everytime I ctrl + c to stop the server, controller and model won't stop actually, it need kill -9 the process.
That's quite trivial to implement in a shell script - make one that launches the three of them in the background, and a callback to kill them when the script itself is killed.
Something like this?
#!/bin/bash
# Launch three processes in the background
process1 &
pids+=($!)
process2 &
pids+=($!)
process3 &
pids+=($!)
# Trap the TERM signal and kill the background processes
trap 'kill ${pids[@]}' TERM
# Wait for any child processes to complete
wait
It's a shell script. Teaching elementary terminal skills is not in the scope of the bug, but let's go.
Have a look at something like https://towardsdatascience.com/basics-of-bash-for-beginners-92e53a4c117a
@surak I mean, I don't know how to define such process1, how to send it as shell args?
Replace line process 1 with the call for the controller, process2 with the web server, process 3 with the worker.
Just make sure you keep the ampersand & at the end of the line.
Google Colab redirects localhost to the VM backend, using 127.0.0.1 solves the issue:
!python3 -m fastchat.serve.controller --host 127.0.0.1 --port 8000
