Question about the metric reported in the paper?

[dsj96]

Question about the metric reported in the paper?. HELLO! I am a new NLPer. I am confused about the pipline(pretrain->fineturn->test) of pre-training large language models.

1. I would like to know which stage of the model was used for unlabeled dataset (e.g., WebText), labeled dataset (e.g., LAMBADA, CoQA, CNN and Daily Mail dataset), respectively? Dose GPT2 model pre-trained on unlabeled dataset, then fine-tuned on labeled dataset (e.g., LAMBADA, CoQA, CNN and Daily Mail dataset), respectively? Finally, reported the score in the paper.
2. Other Large Language Models, like BART, RoBERTa, Mass, have these models been fine-tuned on labeled dataset (e.g., LAMBADA, CoQA, CNN and Daily Mail dataset) before reporting the scores?

Thank you!