Can gradio application run in background ?

[Chesterguan]

trafficstars

Hi I am trying to deploy the model inside a docker container to build up the API service and WebUI :

• base image : I tried both nvidia/cuda:12.1.0-runtime-ubuntu20.04 and nvidia/cuda:11.2.0-runtime-ubuntu18.04
• python version : 3.8
• CUDA: I have both 12.4 and 11.2 version servers

Testing results:

One week ago, I was able to successfully deploy the model in cuda:12.1.0 and connected with controller and built up the API service smoothly.

However, when i re-tried the same operations this week, I cannot successfully register the model worker with controller. I tried either GPUs or CPU. Very weird!!

The commands that I ran:

python3 -m fastchat.serve.controller \
      --host 0.0.0.0 --port 39800 \
        > /tmp/fastChat_CONTROLLER.out \
        2> /tmp/fastChat_CONTROLLER.err & 

sleep 2

python3 -m fastchat.serve.model_worker \
      --host 0.0.0.0 --port 39805 \
      --controller-address http://0.0.0.0:39800 \
      --worker-address http://0.0.0.0:39805 \
      --model-path /tmp/model \
      --model-name 'vicuna-7B' \
      --load-8bit --device cpu \
        > /tmp/fastChat_WORKER.out \
        2> /tmp/fastChat_WORKER.err &

sleep 30

export FASTCHAT_CONTROLLER_URL=http://0.0.0.0:39800

python3 -m fastchat.serve.api \
      --host 0.0.0.0 --port 8900 \
        > /tmp/fastChat_API.out \
        2> /tmp/fastChat_API.err &

sleep 2

echo "FastChat services started."

echo "Start Web UI"

python3 -m fastchat.serve.gradio_web_server \
      --controller-host http://0.0.0.0:39800 \
        > /tmp/fastChat_WEB.out \
        2> /tmp/fastChat_WEB.err &
sleep 5

Please provide some instructions!

[Jeffwan]

@Chesterguan Can you provide some logs from controller side and model worker side?

[Chesterguan]

@Jeffwan Hi Jeff, thanks for asking, and based on my experiments, I figured out the issue now. I could provide more details here.

http://0.0.0.0:model_worker_port is not accessible by controller. The problem comes from Docker container configurations since 0.0.0.0 is non-routable inside containers. might need to add --net=host to make it work.

I believe we could close the issue now.

My current problem is I can successfully run gradio server. It always shows :

[Chesterguan]

Hi I figured out the problems. Seems that the gradio cannot run in background mode ? To reproduce the error, please try: python -m fastchat.serve.gradio_web_server xxxxxx &

[Hzzhang-nlp]

Hi I figured out the problems. Seems that the gradio cannot run in background mode ? To reproduce the error, please try: python -m fastchat.serve.gradio_web_server xxxxxx &

Hello, could you be more specific? For example, how do you generate a public link?

[ZuzooVn]

Hi I figured out the problems. Seems that the gradio cannot run in background mode ? To reproduce the error, please try: python -m fastchat.serve.gradio_web_server xxxxxx &

Did you find the solution?