Dan Saattrup Smart comments

Results 240 comments of


                                            Dan Saattrup Smart

@Mikeriess Thanks! And yep, it caches everything, so will start where it left off 🙂

@Mikeriess That seems to be the 2b-it results? 😮

Thanks @Mikeriess! 🎉 I've updated the leaderboards now. Looks really good 😊

Having the same issue, just with Chrome instead of Chromium.

@larsbun Is it possible to separate the Norwegian/Swedish samples in the dataset, or would we have to use a language identification model?

Can you try manually updating vLLM and trying again?

Seems to be related to [this vLLM issue](https://github.com/vllm-project/vllm/issues/11797).

Doesn't seem like a problem anymore in EuroEval >=15.9.0 (maybe earlier even). Feel free to re-open if it persists.

Ah, I take it back, managed to reproduce it after all. But it only fails on multiple GPUs - a single GPU works just fine.

@Mikeriess Added the Norwegian results now 🙂