unable to communicate with mistral models deployed in EC2 by using gradio_webserver API

[kambalaashok4]

trafficstars

I have deployed mistral7b and 8x&B models in two AWS EC2 server using vllm and fstchat , and now I want to communicate through a web interface(gradio_webserver) and i am using api_endpoints.json file to register models.

api_endpoints.json

{ "Mistral": { "model_name": "mistral7b", "api_type": "openai", "api_base": "http://10.0.0.0/v1", "api_key": "sk-******", "anony_only": false, "recommended_config": { "temperature": 0.7, "top_p": 1.0 }, "text-arena": true, "vision-arena": false, } }

error from interface:

command used to launch webUI

python3 -m fastchat.serve.gradio_web_server --host 0.0.0.0 --port 8080 --controller-url "" --share --register api_endpoints.json