Has anyone successfully reproduced the data in the paper using the code from the evaluation? My results are worse than those reported in the paper

[jianminYa]

🐛 Describe the bug

Has anyone successfully reproduced the data in the paper using the code from the evaluation? My results are worse than those reported in the paper my test results:

I'm not sure what the cause is. Does anyone have any ideas?

[Anand0295]

This bug reflects consistently lower evaluation scores (bleu_score, f1_score, llm_score) than those published in the paper using the same repository code.

Possible cause:

• Environment mismatch (incorrect package/model versions versus original paper)
• Data preprocessing, tokenization, or evaluation scripts have changed or are not matching the original experiment
• Hidden bugs in data loader or scoring function
• Metrics are not using the same formulas/configuration as the paper

Potential solution:

1. Strictly verify environment, package, and model versions against those listed in the paper, including random seed and system setting.
2. Compare the current evaluation/data scripts line-by-line with those used in the paper; revert or update scripts as needed.
3. Request maintainers to post their exact reproducible setup (requirements.txt, dataset hashes, configs, and model checkpoints).
4. Add checks and debug prints to trace scoring logic through each step, validating against published formulas and data samples.
5. After fixing environment and scripts, rerun the experiment and compare outputs with the paper.

Maintainers: Would you be able to share your full reproducible setup so contributors can match the results? Ready to help test once setup details are available.