Follow up on guvectorize use of fastmath

[hammer]

I had to get rid of fastmath=True param from guvectorize due to this issue in Numba.

Originally posted by @aktech in https://github.com/pystatgen/sgkit/issues/306#issuecomment-711432004