FastChat Provider Gives Error Running Commands

[VRImage]

Title

Error running command with provider FastChat

Description

Right after setting a task, as soon as the agent promps the AI through FastChat the following error occurs: ` Executing task 1: Develop a task list.

2023-04-26 22:25:23 2023-04-26 22:25:23 INFO: 172.20.0.1:50156 - "GET /api/agent/Agent-LLM/task/status HTTP/1.1" 200 OK 2023-04-26 22:25:23 Exception in thread Thread-2 (run_task): 2023-04-26 22:25:23 Traceback (most recent call last): 2023-04-26 22:25:23 File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner 2023-04-26 22:25:23 self.run() 2023-04-26 22:25:23 File "/usr/local/lib/python3.10/threading.py", line 953, in run 2023-04-26 22:25:23 self._target(*self._args, **self._kwargs) 2023-04-26 22:25:23 File "/app/AgentLLM.py", line 235, in run_task 2023-04-26 22:25:23 result = self.execution_agent(task["task_name"], task["task_id"]) 2023-04-26 22:25:23 File "/app/AgentLLM.py", line 222, in execution_agent 2023-04-26 22:25:23 return self.run(prompt) 2023-04-26 22:25:23 File "/app/AgentLLM.py", line 80, in run 2023-04-26 22:25:23 self.response = self.instruct(prompt) 2023-04-26 22:25:23 File "/app/provider/fastchat.py", line 15, in instruct 2023-04-26 22:25:23 return response.json()['data'][0].replace("\n", "\n") 2023-04-26 22:25:23 KeyError: 'data' ` Both backend and frontend is running in docker, while FastChat is running in wsl. FastChat is reachable in browser using the URI specified in .env (AI_PROVIDER_URI=http://172.31.111.191:7860/)

Steps to Reproduce

1. Setup FastChat in wsl as specified in documentation for FastChat (https://github.com/lm-sys/FastChat)
2. Edit .env

• AI_PROVIDER=fastchat
• AI_MODEL=vicuna
• AI_TEMPERATURE=0.2
• MAX_TOKENS=2000
• AI_PROVIDER_URI=http://172.31.111.191:7860/

3. Run docker compose up -d
4. Fix issue #82 by running this inside terminal of backend:

apt-get update &&
apt-get install -y --no-install-recommends git build-essential &&
apt-get install g++ -y &&
pip install --upgrade pip &&
pip install --no-cache-dir -r requirements.txt &&
pip install --force-reinstall --no-cache-dir hnswlib &&
apt-get remove -y git build-essential &&
apt-get install libgomp1 -y &&
apt-get install git -y &&
apt-get autoremove -y &&
rm -rf /var/lib/apt/lists/*

5. Run FastChat Controller, Worker and WebServer by running the following script:

IP=$(hostname -I | cut -d' ' -f1) CONTROLLER_PORT=21001 WORKER_PORT=21002

#screen is used to run the commands inside virtual terminals and has no bearing on the commands themselves echo "Running Controller" screen -dmS controller python -m fastchat.serve.controller --host 0.0.0.0 --port ${CONTROLLER_PORT} echo "Running worker" screen -dmS worker1 python -m fastchat.serve.model_worker
--model-path ~/vicuna-7b-1.1
--controller-address http://${IP}:${CONTROLLER_PORT}
--worker-address http://${IP}:${WORKER_PORT}
--host 0.0.0.0
--port ${WORKER_PORT}
--device cpu echo "Waiting 60 seconds" sleep 60 echo "Running webserver" screen -dmS webserver python -m fastchat.serve.gradio_web_server

6. Give a task to an agent.

### Expected Behavior

I would expect that the agent would send a prompt, get a response and build the task list.

### Actual Behavior

The previously noted error occurs.

### Additional Context

_No response_

### Python Version

- [ ] Python <= 3.9
- [X] Python 3.10
- [ ] Python 3.11

### Acknowledgements

- [X] I have searched the existing issues to make sure this bug has not been reported yet
- [X] I am using the latest version of the software
- [X] I have provided enough information for the maintainers to reproduce and diagnose the issue

[VRImage]

After messing around with it a bit, I found that the FastChat API is not part of the webclient, but must be run seperately. I added the following to my FastChat startup script:

API_PORT=8000
export FASTCHAT_CONTROLLER_URL=http://${IP}:${CONTROLLER_PORT}
screen -dmS API python -m fastchat.serve.api --host 0.0.0.0 --port ${API_PORT}

and changed my .env file to reflect this. Though the error generated by the backend did not change, by printing the underlying response from the FastChat API, I could see that it now send it's request to the API and not the web client. The response from the API is now this: {'detail': [{'loc': ['body', 'model'], 'msg': 'field required', 'type': 'value_error.missing'}, {'loc': ['body', 'messages'], 'msg': 'field required', 'type': 'value_error.missing'}]}

EDIT: The does not seem to come from the FastChat API, but from the python requests module.

As a side note, I have tested that the FastChat API works by issuing the following command in a shell:

curl http://172.31.111.191:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vicuna-7b-1.1",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

The response is: {"id":"sXFpPE7QLSdAAfi2pcekPG","object":"chat.completion","created":1682584652,"choices":[{"index":0,"message":{"role":"assistant","content":"Hello! How can I help you today?"},"finish_reason":"stop"}],"usage":null}

[Josh-XT]

https://github.com/Josh-XT/Agent-LLM/blob/main/docs/providers/FASTCHAT.md

Hello! I just updated the FastChat quick start documentation this morning which should help. Please review and make sure your .env file.

AI_PROVIDER=fastchat
AI_MODEL=vicuna-7b-1.1
AI_PROVIDER_URI=http://172.31.111.191:8000