lm-evaluation icon indicating copy to clipboard operation
lm-evaluation copied to clipboard

Published 20 hours ago • verified publisher icon AI21Labs

Integration with lm-eval-harness

Open leogao2 opened this issue 3 years ago • 1 comments

I think it would be great to have AI21 api integration in the EleutherAI lm-eval-harness (https://github.com/EleutherAI/lm-evaluation-harness), which would allow evaluating the AI21 api on far more tasks.

leogao2 avatar Aug 11 '21 14:08 leogao2

Hey! Yes, that would be great! We are still a bit swamped with the launch, but hope to get to it soon.

opherlieber avatar Aug 15 '21 15:08 opherlieber