Dan Saattrup Smart
Dan Saattrup Smart
Support for vLLM has [been merged in now](https://github.com/vllm-project/vllm/pull/15505), so we're just waiting for the next vLLM release now.
Live on the leaderboards now 🎉
This could be a great dataset to add to the benchmark. Whether it should be the default reading comprehension dataset depends on a few points, however: 1. We currently use...
Sure thing. There's still the matter of my second point above. But since the focus of the dataset is on Icelandic culture, it would probably be a great fit for...
@thorunna Fair enough - I can look into it in a few weeks. But note that if we convert it to a multiple choice QA dataset then we should get...
Looks good! We could formulate it as a multiple-choice task with 6 choices, in which case it fits in with the existing tasks. This would fit in the knowledge category...
@usarth Can you try running the evaluation with the `--verbose` and `--raise-errors` flags? It seems like it doesn't recognise that your model is generative properly.
Hi @usarth. I don't have a laptop on me until Wednesday, but one thing that might be wrong is if the pipeline tag hasn't been set to "text-generation". Of course,...
Hi again @usarth. This should be fixed in the newest version now (v13.1.0). In case it still doesn't work on your end then please re-open this issue.
Hi @c-barakat, and thanks for raising the issue. It is definitely _meant_ to work with custom inference APIs, so this is a bug and not a feature 🙂 I think...