MMLU evaluation fails with Mistral

[sanchit-gandhi]

Evaluating any Mistral checkpoint on MMLU throws an error, suggesting that there are several tokens tokens when only one is permitted:

accelerate launch --num_processes=1 run_evals_accelerate.py \
    --model_args "pretrained=hf-internal-testing/tiny-random-MistralForCausalLM" \
    --tasks "original|mmlu|0|0" \
    --override_batch_size 8 \
    --output_dir "./"

Traceback:

Traceback (most recent call last):
  File "/home/sanchit/mistral-evals/run_evals_accelerate.py", line 82, in <module>
    main(args)
  File "/home/sanchit/lighteval/src/lighteval/logging/hierarchical_logger.py", line 166, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/sanchit/lighteval/src/lighteval/main_accelerate.py", line 111, in main
    evaluation_tracker = evaluate(
                         ^^^^^^^^^
  File "/home/sanchit/lighteval/src/lighteval/evaluator.py", line 84, in evaluate
    full_resps = lm.loglikelihood_single_token(requests, override_bs=override_bs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sanchit/lighteval/src/lighteval/models/base_model.py", line 948, in loglikelihood_single_token
    raise ValueError(
ValueError: Trying to do single token multiple choice but one choice has several tokens: [[28705, 330], [28705, 365], [28705, 334], [28705, 384]]. If the additional pre-token is a space, try to set --no_multichoice_continuations_start_space 
Traceback (most recent call last):
  File "/home/sanchit/miniconda3/envs/venv/bin/accelerate", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/sanchit/miniconda3/envs/venv/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 46, in main
    args.func(args)
  File "/home/sanchit/miniconda3/envs/venv/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1075, in launch_command
    simple_launcher(args)
  File "/home/sanchit/miniconda3/envs/venv/lib/python3.11/site-packages/accelerate/commands/launch.py", line 681, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/sanchit/miniconda3/envs/venv/bin/python', 'run_evals_accelerate.py', '--model_args', 'pretrained=hf-internal-testing/tiny-random-MistralForCausalLM', '--tasks', './examples/tasks/subset_cosmo_tasks.txt', '--override_batch_size', '8', '--output_dir', './']' returned non-zero exit status 1.

Adding the suggested flag --no_multichoice_continuations_start_space does not fix the error. For reference: switching to the full mistralai/Mistral-7B-v0.1 checkpoint also yields the same error.

Env:

• transformers version: 4.40.0.dev0
• Platform: Linux-5.4.0-166-generic-x86_64-with-glibc2.31
• Python version: 3.11.8
• Huggingface_hub version: 0.22.2
• Safetensors version: 0.4.2
• Accelerate version: 0.29.2
• Lighteval version: 0.4.0.dev0