cell-dame

hey to make your life easier i already like uploaded the dataset on hugging face https://huggingface.co/datasets/damerajee/khasi-datasets - this is the one split into sentences https://huggingface.co/datasets/damerajee/khasi-raw-data - raw huge paragraphs

> You can limit the #token feed to the model to match the max token length. This is rather a problem with lm_eval_harness than additional tasks. How do we do...

> Thanks for your open sourcing! i'm trying to evaluate `Llama-7b-hf` on `mmlu-fr`, a warning of `Token indices sequence length is longer than the specified maximum sequence length for this...