Unable to reproduce Langboat/ReGPT-125M-200G‘s PPL result.

[dumpmemory]

trafficstars

I have followed the training data setting as discussed in #9 and i used training the model with 200 retrieval index and https://github.com/Langboat/mengzi-retrieval-lm/blob/main/train/config.json. But I can't reproduce the ppl as pretrained model Langboat/ReGPT-125M-200G. Is there any thing i missing ?

Thanks.

[bling0830]

We checked the parameters in config.json, learning_rate should be set to 2e-4.

More detailed model parameters can be found in training_args.bin in huggingface.co/Langboat/ReGPT-125M-200G model file.

Thanks for pointing this out！

[dumpmemory]

Thanks for your reply. I will try again.