Using FastChat UI to Compare Multiple Local LLMs

[soufianechami]

It's not clear in the repo readme how I can use the FastChat UI to compare multiple LLMs on my local machine.

I have these models served via FastAPIs and running on my local server.

Can anyone provide guidance or suggestions on how to set this up effectively?

[jerilkuriakose]

Step 1: Open a new terminal and launch the controller:

python3 -m fastchat.serve.controller

Step 2: Open a new terminal to launch the workers:

python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5

Note: Launch new terminals for each workers

Step 3: Open a new terminal to launch the web server:

python3 -m fastchat.serve.gradio_web_server_multi

The web server will give the URL.