Dan Saattrup Smart

I might have localised the issue now. Can you try reinstalling ScandEval from the `fix/394-qa-error` branch and trying again? You'd need to rebuild your image with ``` pip install scandeval[all]@git+https://github.com/ScandEval/ScandEval@fix/394-qa-error...

> Unfortunately, the issue still persists. I rebuilt the image with the fix branch. But the issue still exists. It occurs after the first iteration of benchmarking and during "Caching...

Also, just updated the branch with (hopefully) a new fix. Try it out 🙂

> Yes, so, I updated the docker file to remove the cache before running scandeval. > > ``` > CMD if [ -z "$args" ]; then \ > echo "No...

Ah, try `/project/.scandeval_cache`

> I deleted .scandeval_cache. But, the issue still exists. It wasn't because of that. > > And I logged out the `self.cache` to see the key value pairs. > >...

Any news on this, @ajude2s?

Closing this issue in that case - feel free to re-open if the problem persists.

The downside is that it only contains 1000 samples. However, these are all _manually annotated_, which is amazing. Maybe it could be a "test-only" dataset, which doesn't allow finetune benchmarking?...

https://github.com/ScandEval/ScandEval/issues/233 seems to be a subset of these.