AgentBench
AgentBench copied to clipboard
THUDM
cg和ltp的std都出现了问题：Error: Worker not responding
ltp出现worker not responding是因为ltp内部需要调openai的api（用turbo做海龟汤的裁判），所以如果没有正确设置openai的key或者连接不稳定或者并发过高等都有可能导致问题。 关于cg可以分享一点更具体的信息吗？如启动的worker数量、评测的模型的情况等。
我的配置如下，麻烦大佬帮忙看下：
openai-chat.yaml
module: src.client.agents.HTTPAgent
parameters:
url: https://api.openai.com/v1/chat/completions
headers:
Content-Type: application/json
Authorization: Bearer sk-___________
body:
temperature: 0
prompter:
name: role_content_dict
args:
agent_role: assistant
return_format: "{response[choices][0][message][content]}"
default.yaml
import: definition.yaml
concurrency:
task:
# dbbench-std: 1
# os-std: 1
# alfworld-std: 1
# webshop-std: 1
# m2w-std: 1
ltp-std: 1
# kg-std: 1
cg-std: 1
agent:
gpt-3.5-turbo-0613: 5
assignments: # List[Assignment] | Assignment
- agent: # "task": List[str] | str , "agent": List[str] | str
- gpt-3.5-turbo-0613
task:
# - dbbench-std
# - os-std
# - alfworld-std
# - webshop-std
# - m2w-std
- ltp-std
# - kg-std
- cg-std
output: "outputs/{TIMESTAMP}"
start_task.yaml
definition:
import: tasks/task_assembly.yaml
start:
# dbbench-std: 1
# os-std: 1
# alfworld-std: 1
# webshop-std: 1
# m2w-std: 1
ltp-std: 1
# kg-std: 1
cg-std: 1
Is your OpenAI key a free trial key? If that's the case, it's important to be aware that free trial keys have rate limits that can be reached quite easily. You can find more information about these rate limits in the OpenAI Documentation. Additionally, the process of evaluating and making LTP calls to the OpenAI API (specifically using the gpt-3.5-turbo model) may lead to faster consumption of your key's usage capacity.
