Dan Saattrup Smart comments

Results 240 comments of


                                            Dan Saattrup Smart

[MODEL EVALUATION REQUEST] nvidia/Hymba-1.5B-Base

Not included in vLLM yet: https://github.com/vllm-project/vllm/issues/10783

[MODEL EVALUATION REQUEST] microsoft/phi-4

@mathiasesn Yep, that should work! Up for the task?

[MODEL EVALUATION REQUEST] microsoft/phi-4

Thanks @mathiasesn, they're live on the leaderboards now 🎉

[MODEL EVALUATION REQUEST] microsoft/phi-4

Thanks @mathiasesn, your initial results are up now 🙂

[MODEL EVALUATION REQUEST] microsoft/phi-4

Live on the leaderboards now 🎉

Feat: Penalize summarization scores for wrong language

@viggo-gascou The only thing that's left here before we can merge are some concrete evaluation tests, to ensure that it doesn't mess up existing evaluations. Can you run the evaluation...

Feat: Penalize summarization scores for wrong language

@viggo-gascou What's the status on this PR?

Feat: Penalize summarization scores for wrong language

@viggo-gascou Closing this PR for now, as it's been stale for a while. Feel free to re-open when the tests have been conducted and it's ready for review.

Support Latvian

The COPA and MMLU translations mentioned in [this paper](https://openreview.net/forum?id=U6Es4V7daa) would be great for reasoning and knowledge evaluations, respectively.

There is a [Latvian citizenship test](https://www.pmlp.gov.lv/en/examinations-determined-citizenship-law?utm_source=https%3A%2F%2Flivelatvia.lv%2F). If previous tests are available then that could be used as a knowledge dataset.