SDMetrics
SDMetrics copied to clipboard
Review GMLogLikelihood metric test noise
The GMLogLikelihood metric was added to cover the metrics that existed in the original SDGym iteration, which used GM Log Likelhood metric over datasets that were simulated using GaussianMixtures.
However, even though the implementation was optimized and improved to make the output as stable and meaningful as possible, the scores produced when this metric is run on datasets which not simulated from GMs tends to be very noisy and may produce inconsistent results between runs. As a consequence of this, the ranking-based integration test fails randomly.
We may want to remove this metric from the ranking test and have a separated one which is tested using GM simulated data, and also add a disclaimer in the documentation indicating what this metric is best suited for.