lm-evaluation-harness icon indicating copy to clipboard operation
lm-evaluation-harness copied to clipboard
verified publisher icon EleutherAI

Chat template fix

Open NathanHB opened this issue 1 year ago • 3 comments

What this PR does

  • Adds a try catch when applying chat template as some models do not support system prompt in their templates.

NathanHB avatar Jul 02 '24 15:07 NathanHB

Unfortunately we cannot have two user messages in a row. What we would need to do is to prepend the description to the first user message.

NathanHB avatar Jul 12 '24 14:07 NathanHB

hi @haileyschoelkopf ! Coming back to this, I was running some tests and it seems like mistral fixed their tokenizer (they now support system prompts). As this fix was made for them I think we can remove it and if a model's tokenizer does not work, it might be up to them to support standard templating.

NathanHB avatar Jul 19 '24 12:07 NathanHB

Hi! Any progress on this? gemma-2-9b-it still requires code editing to evaluate with chat template.

BlackSamorez avatar Sep 11 '24 07:09 BlackSamorez

merged in #2475

baberabb avatar Nov 11 '24 17:11 baberabb

@baberabb I'm still hitting this error using version 0.4.7 , which came out in December 2024 and should have included your PR from November 2024, I believe?

https://github.com/EleutherAI/lm-evaluation-harness/issues/2069#issuecomment-2994459740

Can you please let me know what I should be doing?

RylanSchaeffer avatar Jun 22 '25 21:06 RylanSchaeffer