Numba cache masking compile warnings and deprecations

[jeromekelleher]

Turning off the numba cache seems to cause our CI builds to fail when running tests in NOJIT mode (which we use to get coverage of jitted functions). Seems quite tricky to work around as discussed here: https://github.com/pystatgen/sgkit/pull/1191#issuecomment-1936994466

Related: #1156, #1191

[jeromekelleher]

To reproduce:

NUMBA_DISABLE_JIT=1 python3 -m pytest -v sgkit/tests/test_cohort_numba_fns.py

gives

E   numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)                                                                                                 
E   No implementation of function Function(<built-in function zeros>) found for signature:                                                                                                    
E                                                                                                                                                                                             
E    >>> zeros(int64)                                                                                                                                                                         
E                                                                                                                                                                                             
E   There are 2 candidate implementations:                                                                                                                                                    
E     - Of which 2 did not match due to:                                                                                                                                                      
E     Overload in function 'ol_np_zeros': File: numba/np/arrayobj.py: Line 4315.                                                                                                              
E       With argument(s): '(int64)':                                                                                                                                                          
E      Rejected as the implementation raised a specific error:                                                                                                                                
E        AttributeError: 'function' object has no attribute 'get_call_template'                                                                                                               
E     raised from /home/jk/.local/lib/python3.10/site-packages/numba/core/types/functions.py:541                                                                                              
E                                                                                                                                                                                             
E   During: resolving callee type: Function(<built-in function zeros>)                                                                                                                        
E   During: typing of call at /home/jk/work/github/sgkit/sgkit/stats/cohort_numba_fns.py (191)                                                                                                
E                                                                                                                                                                                             
E                                                                                                                                                                                             
E   File "sgkit/stats/cohort_numba_fns.py", line 191:                                                                                                                                         
E   def cohort_mean(                                                                                                                                                                          
E       <source elided>                                                                                                                                                                       
E       c = len(_)                                                                                                                                                                            
E       count = np.zeros(c)                                                                                                                                                                   
E       ^                                                                                              

[timothymillar]

I have opened https://github.com/numba/numba/issues/9452 related to this. Note that the root cause seems to be a discrepancy between behavior when the jit is/isn't disabled. The caching seems to be hiding the issue because cached (compiled!?) versions seem to be used despite the jit being disabled.

[timothymillar]

The docs suggest that NUMBA_DISABLE_JIT should be ignored by @guvectorize as it only only affects @jit and @njit: https://numba.pydata.org/numba-doc/dev/user/troubleshoot.html#disabling-jit-compilation

[timothymillar]

In the interest of getting things moving, we could:

• Stop running the test suit with NUMBA_DISABLE_JIT=1
• Only use NUMBA_DISABLE_JIT=1 on tests that don't hit guvectorize (e.g., test_pedigree.py)

We're not getting anything from using it for guvectorize functions, which is our main use-case.

[jeromekelleher]

I'm happy to take your lead here @timothymillar, whatever seems like the most pragmatic way forward would be great.

[timothymillar]

I think the way forward is to annotate all of the guvectorize functions with # pragma: no cover again.

[jeromekelleher]

We agreed this is a pragmatic way forward in the meeting today