ShapeWorks icon indicating copy to clipboard operation
ShapeWorks copied to clipboard

Analysis - Adjust generalization and specificity metrics

Open jadie1 opened this issue 11 months ago • 0 comments

Currently, the generalization is measured using the distance between true and reconstructed unseen particles, and specificity is measured using the distance between sampled particles and closest real particles.

The problem with this is it is dependent on the accuracy of correspondence. When the SSM has miscorrespondences they may be incorrectly captured in PCA as variations that are actually present in the training data. This can lead to a model that, when generating new samples, produces shapes that are closer to these "noisy" versions of the training shapes, yielding good specificity when the sampled shapes do not actually resemble the shape population.

To remedy this, generalization and specificity should not be quantified as distance between particles, but rather as distance between particles and the ground truth shape/surface. Meaning generalization is the distance between reconstructed particles and the true shape surface and specificity is the distance between sampled particles and the surface. This would provide more consistent metrics.

jadie1 avatar Mar 05 '24 20:03 jadie1