llm-foundry icon indicating copy to clipboard operation
llm-foundry copied to clipboard

Published 20 hours ago • verified publisher icon mosaicml

HumanEval Benchmark

Open L1aoXingyu opened this issue 1 year ago • 2 comments

🚀 Feature Request

I found there is only ICL Benchmark in eval folder, but the HumanEval Benchmark is reported in MPT-30B. I want to reproduce the HumanEval results with llm-foundry.

So I just want llm-foundry integrated with HumanEval Benchmark.

Motivation

[Optional] Implementation

Additional context

L1aoXingyu avatar Jun 23 '23 13:06 L1aoXingyu

Hi! We are working on integrating HumanEval into the current coding suite. Thank you for your patience while we do so :)

bmosaicml avatar Jun 23 '23 14:06 bmosaicml

@bmosaicml hi, I want to train codeLLM using llm-foundry, any tutorial or docs about it? Thank you so much!

L1aoXingyu avatar Jun 27 '23 05:06 L1aoXingyu

This has been merged in #587

dakinggg avatar Oct 05 '23 07:10 dakinggg