Surprise icon indicating copy to clipboard operation
Surprise copied to clipboard

Potential numerical instability in pearson sim

Open NicolasHug opened this issue 7 years ago • 2 comments

Following #214, there might be some numerical instability in pearson similarity computation, probably caused by the sqrt function receiving either negative values, NaN or infinity.

I couldn't reproduce the issue myself.

NicolasHug avatar Oct 15 '18 01:10 NicolasHug

Also from #224, there seem to be some pretty big differences between the current computation and the one with the classical method

NicolasHug avatar Nov 09 '18 20:11 NicolasHug

Also from #224, there seem to be some pretty big differences between the current computation and the one with the classical method

Update: actually no, all is fine on this side. I had made an error in pearson_std_formula: the means should have been computed on the common items only: that's what we're doing in surprise. There's actually a comment about whether or not this should be done in Aggarwal textbook: the one in surprise is the strictly correct one, even though it's not clear whether it makes a big difference.

NicolasHug avatar Dec 09 '18 16:12 NicolasHug