pipilia

[MiniLLM] problem in evaluate() function

1

I attempted to apply the `evaluate()` method from your `finetune.py` to the training process of MiniLLM, but I noticed that `model.eval()` causes anomalies in the `rl_loss`. Specifically, at certain fixed...

[MiniLLM] roberta dataset differences between llama version and opt version

4

Hi , I found that in the link you provide ([https://huggingface.co/datasets/MiniLLM/roberta-corpus-processed](url)) there is a huge size gap between 'llama/512/20M/train_0.bin'(200MB) and 'opt/512/20M/train_0.idx'(20.5GB) . And I do suffer a Traceback that says...

In your Readme , I noticed that you said the LM corpus (like openwebtext in gpt2 and roberta in Llama ) is unnecessary in your method.Do you consider sharing the...