setigen
setigen copied to clipboard
Computation of noise stats should use `dtype=np.float64`
Computing the mean and std of large frames (e.g. ~16 million points) can suffer from floating point error when they are performed using float32 values. Both mean and std can be passed dtype=np.float64 to perform these computations using float64 instead and thereby retain better accuracy. One place where this could be added is here:
https://github.com/bbrzycki/setigen/blob/5ca049f80b44a9dd9bffc8889d46d2135f4e4011/setigen/frame.py#L398
but perhaps it would be good to add a dtype=np.float64 parameter to the enclosing function so that the user could specify a different type if desired.
There may be other places where this change could be applicable as well.