Can not reproduce the bart-large results on xsum dataset

[szhang42]

Hello,

I'm trying to reproduce the results of BART on XSUM dataset.

I followed README, didn't apply any preprocessing to the XSUM data and use beam=6, lenpen=1.0, max_len_b=60, min_len=10 for generation (following this link https://github.com/facebookresearch/fairseq/blob/main/examples/bart/README.summarization.md#4-fine-tuning-on-cnn-dm-summarization-task). I also use the provided the model checkpoint from fairseq at this link (https://github.com/facebookresearch/fairseq/tree/main/examples/bart) for bart large.

I got the following results. Could anyone advise on this and what should I change to achieve the results reported in the paper as 45.14 22.27 37.25?

1 ROUGE-1 Average_F: 0.44189 (95%-conf.int. 0.43897 - 0.44452) 1 ROUGE-2 Average_F: 0.20829 (95%-conf.int. 0.20558 - 0.21104) 1 ROUGE-L Average_F: 0.35156 (95%-conf.int. 0.34865 - 0.35432)

Thanks very much!