Koan-Sin Tan

@farook-edev: for quantized models, it could generate repeating tailing patterns. @freedomtan: it appears to a common issue, I'll ask around to see if we can get rid of that.

This phenomenon is called Neural Text Degeneration. There are many articles and papers discussing it. And generally, quantization could magnify Neural Text Degeneration. > [@farook-edev](https://github.com/farook-edev): for quantized models, it could...

@freedomtan to verify @farook-edev's findings. If confirmed, we should report to Google guys' (send a PR / issue ). @farook-edev to check the 33 prompts provided by @AhmedTElthakeb for the...

> [@freedomtan](https://github.com/freedomtan) to verify [@farook-edev](https://github.com/farook-edev)'s findings. If confirmed, we should report to Google guys' (send a PR / issue ). > > [@farook-edev](https://github.com/farook-edev) to check the 33 prompts provided by...

> [@freedomtan](https://github.com/freedomtan) That's very curious.. I redownloaded the [python code](https://github.com/google-research/google-research/tree/master/instruction_following_eval) and re-ran the strict comparison with python3.10 and python3.13 > > The difference I got from my original google test...

we can use the mobile_open

language stemming: convert plurals, -ing back to the original form. war, wars: twice of war warrant will be counted in google's implementation. strstr() in C current: exact match now. strict...

@farook-edev let's test performance model with the 33 prompts from ifeval with early stoping. conditions: min_time, min_samples (default value = 1024?, let's make it to be 5), max_running (early stoping)

if we can "group" the two datasets for one LLM (1b or 8b). That's good.

@freedomtan to check if the running order (offline one should be the latest one to run) is kept unchanged.