setigen icon indicating copy to clipboard operation
setigen copied to clipboard

Computation of noise stats should use `dtype=np.float64`

Open david-macmahon opened this issue 2 years ago • 1 comments

Computing the mean and std of large frames (e.g. ~16 million points) can suffer from floating point error when they are performed using float32 values. Both mean and std can be passed dtype=np.float64 to perform these computations using float64 instead and thereby retain better accuracy. One place where this could be added is here:

https://github.com/bbrzycki/setigen/blob/5ca049f80b44a9dd9bffc8889d46d2135f4e4011/setigen/frame.py#L398

but perhaps it would be good to add a dtype=np.float64 parameter to the enclosing function so that the user could specify a different type if desired.

david-macmahon avatar Feb 24 '23 02:02 david-macmahon

There may be other places where this change could be applicable as well.

david-macmahon avatar Feb 24 '23 05:02 david-macmahon