Compare loss on enwik8

[Bachstelze]

The validation loss went to 1.3564 after 1 epoch of training with a small model on enwik8. How can this be evaluated and compared with respect to other models?

[conceptofmind]

@Bachstelze That is difficult to properly evaluate at a smaller scale without sufficient pre-training. It would likely require a single go over something like OpenWebText or Wikitext if you wanted to get a more detailed analysis. ColossalAI trained a PaLM model based on Lucid's repository on Wikitext and it may be worth checking out their results: https://github.com/hpcaitech/PaLM-colossalai