mengzi-retrieval-lm icon indicating copy to clipboard operation
mengzi-retrieval-lm copied to clipboard

Published 20 hours ago • verified publisher icon Langboat

Unable to reproduce PPL for GPT-Neo-125M using lm-eval

Open pedrogengo opened this issue 2 years ago • 1 comments
trafficstars

Hey!

I'm trying to run the following command using the lm-eval cli, but I can't reproduce the results you shared. Did you do something different? If not, do you have any idea where I'm doing wrong?

python main.py \
	--model gpt2 \
	--model_args pretrained=EleutherAI/gpt-neo-125M \
	--device 0 \
	--tasks wikitext \
	--batch_size 1

pedrogengo avatar Dec 13 '22 21:12 pedrogengo

May I ask what is the result of the EleutherAI/gpt-neo-125? Did you get an error when using it?

bling0830 avatar Dec 27 '22 05:12 bling0830