falcon icon indicating copy to clipboard operation
falcon copied to clipboard

Published 20 hours ago • verified publisher icon bittremieux

Weird numba caching issue

Open tycheyoung opened this issue 2 years ago • 2 comments

Due to the incorrect fraction part, the resulting dim can have an error (+1/-1 difference).

tycheyoung avatar Mar 09 '22 18:03 tycheyoung

Could you elaborate a bit? When exactly does the issue occur and is it related to Numba or just a general issue?

bittremieux avatar Mar 10 '22 09:03 bittremieux

If we set min_mz = 101.00, max_mz = 1500.00, bin_size=0.05, the dimension needs to be 27981. When we enable numba caching, the fraction part of bin_size changes (in my system, it changes to 0.05000000074505806), resulting in 27982 dimension.

tycheyoung avatar Mar 10 '22 14:03 tycheyoung