eval_loss结果与原文结果不一致

[APiaoG]

您好！我在复现论文eval_loss的结果时发现评估结果不一致，我想问一下这是正常的吗？ 以下是原文结果 我自己复现的结果如下：

[APiaoG]

我猜测这可能与gsm8k 训练集和测试集传入eval_loss的方式有关，我采用的是question + ‘ ’ + answer的方式构造类似Skywork/mock_gsm8k_test数据集的text传入进行计算，我想问一下您实现的时候是如何构造的呢？

[zhao1iang]

我们也是直接拼接起来的，和您的拼接方式似乎没有什么区别，下面是两条我们拼接的样例。 {"text": "Tina makes $18.00 an hour. If she works more than 8 hours per shift, she is eligible for overtime, which is paid by your hourly wage + 1/2 your hourly wage. If she works 10 hours every day for 5 days, how much money does she make? She works 8 hours a day for $18 per hour so she makes 818 = $<<818=144.00>>144.00 per 8-hour shift\nShe works 10 hours a day and anything over 8 hours is eligible for overtime, so she gets 10-8 = <<10-8=2>>2 hours of overtime\nOvertime is calculated as time and a half so and she makes $18/hour so her overtime pay is 18*.5 = $<<18*.5=9.00>>9.00\nHer overtime pay is 18+9 = $<<18+9=27.00>>27.00\nHer base pay is $144.00 per 8-hour shift and she works 5 days and makes 5 * $144 = $<<1445=720.00>>720.00\nHer overtime pay is $27.00 per hour and she works 2 hours of overtime per day and makes 272 = $<<272=54.00>>54.00 in overtime pay\n2 hours of overtime pay for 5 days means she makes 545 = $270.00\nIn 5 days her base pay is $720.00 and she makes $270.00 in overtime pay so she makes $720 + $270 = $<<720+270=990.00>>990.00\n#### 990"} {"text": "A deep-sea monster rises from the waters once every hundred years to feast on a ship and sate its hunger. Over three hundred years, it has consumed 847 people. Ships have been built larger over time, so each new ship has twice as many people as the last ship. How many people were on the ship the monster ate in the first hundred years? Let S be the number of people on the first hundred years\u2019 ship.\nThe second hundred years\u2019 ship had twice as many as the first, so it had 2S people.\nThe third hundred years\u2019 ship had twice as many as the second, so it had 2 * 2S = <<2*2=4>>4S people.\nAll the ships had S + 2S + 4S = 7S = 847 people.\nThus, the ship that the monster ate in the first hundred years had S = 847 / 7 = <<847/7=121>>121 people on it.\n#### 121"}

注意到您计算的ref loss和我们是差不多的，应该不是脚本的问题，我附上我拼接的gsm8k_train和gsm8k_test您可再比较一下。 gsm8k_train_test.zip

[APiaoG]

我们也是直接拼接起来的，和您的拼接方式似乎没有什么区别，下面是两条我们拼接的样例。 {"text": "Tina makes $18.00 an hour. If she works more than 8 hours per shift, she is eligible for overtime, which is paid by your hourly wage + 1/2 your hourly wage. If she works 10 hours every day for 5 days, how much money does she make? She works 8 hours a day for $18 per hour so she makes 8_18 = $<<8_18=144.00>>144.00 per 8-hour shift\nShe works 10 hours a day and anything over 8 hours is eligible for overtime, so she gets 10-8 = <<10-8=2>>2 hours of overtime\nOvertime is calculated as time and a half so and she makes 18/hoursoherovertimepayis18∗.5=<<18*.5=9.00>>9.00\nHer overtime pay is 18+9 = $<<18+9=27.00>>27.00\nHer base pay is $144.00 per 8-hour shift and she works 5 days and makes 5 * 144=<<144_5=720.00>>720.00\nHer overtime pay is $27.00 per hour and she works 2 hours of overtime per day and makes 27_2 = $<<27_2=54.00>>54.00 in overtime pay\n2 hours of overtime pay for 5 days means she makes 54_5 = $270.00\nIn 5 days her base pay is $720.00 and she makes $270.00 in overtime pay so she makes $720 + 270=<<720+270=990.00>>990.00\n#### 990"} {"text": "A deep-sea monster rises from the waters once every hundred years to feast on a ship and sate its hunger. Over three hundred years, it has consumed 847 people. Ships have been built larger over time, so each new ship has twice as many people as the last ship. How many people were on the ship the monster ate in the first hundred years? Let S be the number of people on the first hundred years\u2019 ship.\nThe second hundred years\u2019 ship had twice as many as the first, so it had 2S people.\nThe third hundred years\u2019 ship had twice as many as the second, so it had 2 * 2S = <<2*2=4>>4S people.\nAll the ships had S + 2S + 4S = 7S = 847 people.\nThus, the ship that the monster ate in the first hundred years had S = 847 / 7 = <<847/7=121>>121 people on it.\n#### 121"}

注意到您计算的ref loss和我们是差不多的，应该不是脚本的问题，我附上我拼接的gsm8k_train和gsm8k_test您可再比较一下。 gsm8k_train_test.zip

非常感谢您的答复！我使用了您的数据进行了复现，还是得到了我上述图片中的结果，您对此有什么解决的方法吗？非常感谢！