Unusual method for population re-sampling in bootstrap

[dennlinger]

Hi, great work and really interesting approach to NLG evaluation!
I was going through your implementation of computing paired bootstrap tests for estimating the significance of results and found an unusual way with the re-sampling of your population estimate, e.g., the one below (other testing functions perform similar sampling in the same script):

https://github.com/neulab/BARTScore/blob/248f511cb34ae3753fc81f7d7a945de5bfe33458/analysis.py#L104

According to literature, bootstrap sampling generally performs sampling with replacement to the size of the original data, see, e.g., (Koehn, 2004). I am not sure how much this affects significance, but especially for the rather small QAGS datasets this might have some effect. Did you follow any recommendations when choosing the portion to sub-sample to (i.e., the 80%), or is this a more or less arbitrarily set threshold?

Thanks in advance for any insights! Best, Dennis