Jack Morris

[in progress] updating libraries. datasets 2.4, transformers 4.2, gensim 4

And some minor code cleanups.

This isn't really an issue per se, but I found that if you wrap the entire call `grad_cache.GradCache(...)` in torch autocast, you will run into weird errors. This happens by...

When training models, the bulk of evaluation is done on the main worker. When we train with 8 GPUs, we should get around an 8x speedup on eval, which would...

Tokenizer alignment code

1

Hi, I don't see any code in the repository for performing alignment between tokenizers. It would be really helpful if you could share this portion of your code.

I see this error all the time and it was starting to get to me.

Fix caching logic

4

Right now the code throws an error if you try to evaluate the same dataset with different numbers of shots. My PR fixes that.

[BUG] FileNotFoundError with newest lighteval 0.11.0

2

## Describe the bug I receive a FileNotFoundError: ``` │ /home/jxm/anaconda3/lib/python3.12/site-packages/datasets/load.py:1033 in dataset_module_factory │ │ │ │ 1030 │ │ │ │ │ ) from None │ │ 1031 │...

[BUG] TriviaQA numbers are off

4

## Describe the bug I tried running triviaQA through lighteval and all the numbers seem to be zero, e.g. ```bash $ lighteval vllm "model_name=Qwen/Qwen3-14B-AWQ" "lighteval|triviaqa|0|0" ... [2025-08-04 15:42:37,715] [ INFO]:...