congyingxia

Thanks a lot, I can get similar results right now: Eval metrics/winograd/0-shot/InContextLearningMultipleChoiceAccuracy: 0.8679 The config has some space issues, a fixed version is provided here: ``` seed: 1 max_seq_len: 1024...

Do you have any idea why there is such a large difference between the results obtained from running 'python eval/eval.py' and 'composer eval/eval.py'?

I tried both multi-gpu and single-gpu for "python eval.py". The issue is the same. So it should not due to the setup of using multi-gpu.

Thanks for your clarification. Is this the reason for the difference between running 'python eval/eval.py' and 'composer eval/eval.py'? python eval/eval.py is using the exact samples in the dataset while composer...

Got it, thanks!