replicate the results for forward synthesis on the USPTO-MIT dataset

[feiyang-cai]

Hello,

Thanks for your nice work.

I am recently encountering some issues with replicating the results for forward synthesis on the USPTO-MIT dataset in the mixed setting.

According to the tutorial, I downloaded the dataset and the trained model, and ran:

1. onmt_translate -config train-from-scratch/RtoP/RtoP-MIT-mixed-aug5-translate.yml
2. python ./score.py -beam_size 10 -n_best 10 -augmentation 5 -targets ./dataset/USPTO-MIT_RtoP_aug5_mixed/test/tgt-test.txt -predictions ./exp/USPTO-MIT_RtoP_aug5_mixed/average_model_196-200-results.txt -process_number 8 -score_alpha 1 -save_file ./final_results.txt -source ./dataset/USPTO-MIT_RtoP_aug5_mixed/test/src-test.txt

The result is Top-1 Acc:89.972%, MaxFrag 91.450%, Invalid SMILES:0.103% Sorted Invalid SMILES:0.000% Top-2 Acc:94.505%, MaxFrag 95.073%, Invalid SMILES:12.698% Sorted Invalid SMILES:0.000% Top-3 Acc:95.568%, MaxFrag 96.057%, Invalid SMILES:18.905% Sorted Invalid SMILES:0.000% Top-4 Acc:96.100%, MaxFrag 96.532%, Invalid SMILES:22.236% Sorted Invalid SMILES:0.002% Top-5 Acc:96.433%, MaxFrag 96.835%, Invalid SMILES:24.846% Sorted Invalid SMILES:0.003% Top-6 Acc:96.642%, MaxFrag 97.030%, Invalid SMILES:26.697% Sorted Invalid SMILES:0.011% Top-7 Acc:96.847%, MaxFrag 97.220%, Invalid SMILES:28.962% Sorted Invalid SMILES:0.024% Top-8 Acc:96.955%, MaxFrag 97.330%, Invalid SMILES:32.426% Sorted Invalid SMILES:0.060% Top-9 Acc:97.067%, MaxFrag 97.447%, Invalid SMILES:37.722% Sorted Invalid SMILES:0.120% Top-10 Acc:97.170%, MaxFrag 97.552%, Invalid SMILES:46.771% Sorted Invalid SMILES:0.221% Unique Rates:99.780% It's around 1% less than the value reported in the paper.

I didn't do any training and just ran based on the model uploaded, so I am wondering if the model was uploaded by mistake or any configuration I mistakenly used.

Thanks!