webarena icon indicating copy to clipboard operation
webarena copied to clipboard

Wrong scroll action specification in the prompts vs. in the parsing function

Open michalspiegel opened this issue 2 years ago • 2 comments

Hi, I'm reporting just a small bug, the prompts all say to use "scroll [direction=up|down]" but the parsing function expects "scroll [up|down]". This way the parsing always fails for the scroll action because the model always generates "scroll [direction=up|down]".

michalspiegel avatar Mar 13 '24 20:03 michalspiegel

@shuyanzhou

frankxu2004 avatar Apr 15 '24 05:04 frankxu2004

On the other hand, judging from you GPT4 experiment logs, this was not a problem for GPT4. Maybe this is just specific for the Gemini I was using. It always tried to generate the action in the wrong format, e.g. scroll [direction=down]. In that case this issue might be irrelevant

michalspiegel avatar Apr 18 '24 15:04 michalspiegel