alfworld的成功率只有0.09

[XGJ111]

我使用提供的AgentEvol-7B模型，把max_round设置为"30"，使用base_eval_script.sh进行测试，得到的结果如下

[carrot-codeing]

一样的情况，能帮忙看看这个问题吗 @WooooDyy

[realtmxi]

same case @WooooDyy

[MHlk]

same case @WooooDyy , the score of alfworld seems not align with the paper

[Andy15]

Hi, @XGJ111 @carrot-codeing @realtmxi @MHlk .

Thanks for your comments and feedback. We checked our evalutaion process as well as our ALFWorld implementations, and we found that the cause lies in the updated version of the original ALFWorld library.

We suggest to use alfworld==0.3.3 in AgentGym. Note that you may need to remove the game files of newer ALFWorld lib (e.g. rm -rf ~/.cache/alfworld by default) and re-run alfworld-download.