MonteCarloMeasurements.jl
MonteCarloMeasurements.jl copied to clipboard
Use estimated quantiles instead of std
Similar to #26 Instead of reducing particles to uncertainty by calling std, we can make use of the quantiles of the sample, at least if number of particles is large. Example code: https://github.com/baggepinnen/MonteCarloMeasurements.jl/blob/5c4d1383405fe3cc013eb4a54d35071e4d013460/src/particles.jl#L385
Asking @cscherrer for input here: I currently have definitions like the following to determine if particles differ significantly from each other or from a scalar
Base.:≈(a::Real,p::AbstractParticles, lim=2) = abs(mean(p)-a)/std(p) < lim
≲(p::AbstractParticles,a::AbstractParticles,lim=2) = (mean(p)-mean(a))/(2sqrt(std(p)^2 + std(a)^2)) > lim
but I guess it would make sense to change these to using sample quantiles instead, at least if the number of particles is reasonably large (might not make sense for sigmapoints
etc.). E.g., the following definition
Base.:≈(a::Real,p::AbstractParticles, q=0.025) = quantile(p,q) < a < quantile(p,1-q)
I think this makes sense, but am open to hear your thoughts on it
Oh this is tricky - I think it really depends on what use case you're targeting.
If you're considering particles as proxies for random variables, I'd think the ordering matters (since permuted particles are no longer equal). So I'd probably go with "difference is not statistically significant", so maybe something like
function Base.:≈(a::AbstractParticles, b)
lo,hi = quantile(a-b, (0.05,0.95))
lo < 0.0 < hi
end
EDIT: I just realized this is the same as your example. Been a long day :)
If you want to consider them only as distributions, then I'd probably go with the Wasserstein or Earth mover distance
Distribution free test statistic to determine if two samples come frome the same distribution.
A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Scholkopf, and A. Smola. ¨ A kernel two-sample test.