webarena

webarena copied to clipboard

Overuse of exact matches in eval.

7

I would like to re-open issue #104 There's an overuse of exact matches in the eval harness. For example, consider task 649: ``` "intent": "Post in history subreddit about what...

afourney

Issue with the reddit website?

8

I am not super sure but I noticed something weird with the reddit website in the benchmark: I was looking at task 29: "Tell me the count of comments that...

gagb

Unstablility of gitlab website

1

Dear Authors, I deployed the Gitlab website locally, but I found that it is unstable, e.g., if I keep visiting it, sometimes it works normally, while sometimes it reports 502....

taogoddd

Pop-up windows and Combobox can not be viewed

1

Dear authors, Thanks for your nice work and quick follow-ups for issues. I found two problems when using the benchmark, especially on the map website: 1. The pop-up windows, e.g.,...

taogoddd

Typo in task template 19

2

There's a typo in task template 19 (noticed for task 649). The template is: "Post in {{subreddit}} subreddit about what could diffusion model help the **correpong** field" It should be...

afourney

Sorting issue for task id 49

1

The gold URL for task #49 contains a bug. ``` { "sites": [ "gitlab" ], "task_id": 45, "require_login": true, "storage_state": "./.auth/gitlab_state.json", "start_url": "__GITLAB__/a11yproject/a11yproject.com", "geolocation": null, "intent_template": "Check out the most...

afourney

Hi, there. I am working on some map tasks and have found it hard to accomplish them even by myself. The usage of the OpenStreetMap is quite different from the...

zijianma17

Environments are 300GB+

1

I am wondering why the docker tar files are so large. I am trying to reproduce the environment locally to avoid paying the EC2 pricing.

devinat1

Wrong scroll action specification in the prompts vs. in the parsing function

2

Hi, I'm reporting just a small bug, the prompts all say to use "scroll [direction=up|down]" but the parsing function expects "scroll [up|down]". This way the parsing always fails for the...

michalspiegel

Support for HF models

2

Hi, do we support huggingface models? If any internal internal tests were done with llama and the insights can be shared that will be great

karthik19967829