torsional-diffusion icon indicating copy to clipboard operation
torsional-diffusion copied to clipboard

the result fairness?

Open wang1215789 opened this issue 3 years ago • 2 comments

when you compare your model result with others, the result is far beyond baselines.But, your model is train on a different train data, does it is fair to other models? In your ablation experiments, your model trained on true conformations almost same as others, even worse than others. image image

wang1215789 avatar Sep 30 '22 02:09 wang1215789

Dear @wang1215789 I think there may be a few levels of miscomprehension:

  1. The other machine learning baselines (GeoMol and GeoDiff) are trained with the exact same data. On the other hand, not having access to RDKit and OMEGA parameter estimation routines we could not retrain those with the same data.
  2. Training on ground truth L does not work well as expected because of the mismatch in the local structures that we discussed in the paper. However, that does not mean that we do not train on ground truth conformations, we simply train our model with conformers with local structures from RDKit that resemble as much as possible those of the ground truth. Training other models with these perturbed conformers would just make the results worst.

Is there anything that I missed in your questions? I hope this clarifies your concerns.

gcorso avatar Sep 30 '22 18:09 gcorso

Did you Training other models (GeoMol and GeoDiff) with your perturbed conformers ? The result was worest than before? I think the dataset used now may be biased and the evaluation(recall and precision) used may be not good to judge a model, but we dont have another one. Thanks for your reply. Your job is a great work for 3D molecule conformer generation.

wang12-github avatar Oct 05 '22 04:10 wang12-github

No, we did not use the perturbed conformers to train the other methods, but we don't see any reason why it would help them. When we test we use the ground truth, not the perturbed conformers, therefore, I'm not sure what bias would be present in the evaluation. Thanks, Gabriele

gcorso avatar Nov 16 '22 19:11 gcorso