Dan Saattrup Nielsen
Dan Saattrup Nielsen
### 🐛 Describe the bug When benchmarking `intfloat/multilingual-e5-large-instruct` we encounter the following error: `scandeval.exceptions.InvalidBenchmark: NaN value detected in model outputs, even with mixed precision disabled.` If we force the dtype...
### Model ID state-spaces/mamba-2.8b-hf ### Model type State space model (e.g., Mamba) ### Model languages - [ ] Danish - [ ] Swedish - [ ] Norwegian (Bokmål or Nynorsk)...
### Model ID mistralai/Mixtral-8x7B-instruct-v0.1 ### Model type Decoder model (e.g., GPT) ### Model languages - [X] Danish - [X] Swedish - [X] Norwegian (Bokmål or Nynorsk) - [X] Icelandic -...
### Model ID mistralai/Mixtral-8x7B-v0.1 ### Model type Decoder model (e.g., GPT) ### Model languages - [X] Danish - [X] Swedish - [X] Norwegian (Bokmål or Nynorsk) - [X] Icelandic -...
### Model ID google/gemma-7b-it ### Model type Decoder model (e.g., GPT) ### Model languages - [ ] Danish - [ ] Swedish - [ ] Norwegian (Bokmål or Nynorsk) -...
### Model ID google/gemma-7b ### Model type Decoder model (e.g., GPT) ### Model languages - [ ] Danish - [ ] Swedish - [ ] Norwegian (Bokmål or Nynorsk) -...
Since we added the new Danish knowledge datasets Danske Talemåder and Danish Citizen Tests, we need to evaluate existing leaderboard models on these. This has been done for 7B models...
### Model ID DiscoResearch/DiscoLM-mixtral-8x7b-v2 ### Model type Decoder model (e.g., GPT) ### Model languages - [ ] Danish - [ ] Swedish - [ ] Norwegian (Bokmål or Nynorsk) -...
### Model ID seedboxai/KafkaLM-70B-German-V0.1 ### Model type Decoder model (e.g., GPT) ### Model languages - [ ] Danish - [ ] Swedish - [ ] Norwegian (Bokmål or Nynorsk) -...
### Model ID seedboxai/KafkaLM-13B-German-V0.1-DPO ### Model type Decoder model (e.g., GPT) ### Model languages - [ ] Danish - [ ] Swedish - [ ] Norwegian (Bokmål or Nynorsk) -...