aeon
aeon copied to clipboard
aeon-toolkit
[BUG] Some tests fail without numba
Describe the bug
When numba is disabled using NUMBA_DISABLE_JIT=1 some tests begin failing. We disable numba for our code coverage test, as it does not play well together currently (https://github.com/numba/numba/issues/4268).
Failing tests: see #625 #626
Example nightly test run: https://github.com/aeon-toolkit/aeon/actions/runs/5721839977/job/15504019987
Steps/Code to reproduce the bug
Run the test suite with the NUMBA_DISABLE_JIT environment variable set to 1.
Expected results
No unexpected tests fail even with numba disabled.
Actual results
Tests which normally pass start failing.
Versions
N/A, see CI run
having looked into #1498 and found a bug that numba was hiding, it might be good too look at these failures in more detail, so will list here
- [ ] test_all_classifiers:test_classifier_against_expected_results HIVECOTEV2 fails
def test_classifier_against_expected_results(self, estimator_class):
"""Test classifier against stored results."""
# we only use the first estimator instance for testing
classname = estimator_class.__name__
# the test currently fails when numba is disabled. See issue #622
import os
if classname == "HIVECOTEV2" and os.environ.get("NUMBA_DISABLE_JIT") == "1":
return None
-
[ ] StatsForecastAutoARIMA #625 this is excluded from all tests in config if numba is not used
-
[ ] test_catch22_on_basic_motions(): excluded
def test_catch22_on_basic_motions():
"""Test of Catch22 on basic motions data."""
# the test currently fails when numba is disabled. See issue #622
import os
if os.environ.get("NUMBA_DISABLE_JIT") == "1":
return None
so I think both fails are caused by catch22, and by a tedious process of elimination I have isolated the problem to this function. Error happens when you comment out the njit
@staticmethod
@njit(fastmath=True, cache=True)
def _SB_MotifThree_quantile_hh(X):
# Shannon entropy of two successive letters in equiprobable 3-letter
# symbolization.
indicies = np.argsort(X)
bins = np.zeros(len(X))
q1 = int(len(X) / 3)
q2 = q1 * 2
l1 = np.zeros(q1, dtype=np.int_)
for i in range(q1):
l1[i] = indicies[i]
l2 = np.zeros(q1, dtype=np.int_)
c1 = 0
for i in range(q1, q2):
bins[indicies[i]] = 1
l2[c1] = indicies[i]
c1 += 1
l3 = np.zeros(len(indicies) - q2, dtype=np.int_)
c2 = 0
for i in range(q2, len(indicies)):
bins[indicies[i]] = 2
l3[c2] = indicies[i]
c2 += 1
found_last = False
nsum = 0
for i in range(3):
if i == 0:
o = l1
elif i == 1:
o = l2
else:
o = l3
if not found_last:
for n in range(len(o)):
if o[n] == len(X) - 1:
o = np.delete(o, n)
break
for n in range(3):
nsum2 = 0
for v in o:
if bins[v + 1] == n:
nsum2 += 1
if nsum2 > 0:
nsum2 /= len(X) - 1
nsum += nsum2 * np.log(nsum2)
return -nsum
ok, I have narrowed this down to this operation
if nsum2 > 0:
nsum2 /= len(X) - 1
nsum += nsum2 * np.log(nsum2)
basically with numba turned on, 19/99
Numba = 0.19191919191919193 given to 17dp)
whereas with numba turned off we get
No numba = 0.1919191919191919 (16 dp)
even if I round number to 16 dp, the error comes through np.log. So really, three options
- leave as is with this bug issue
- remove njit from this function
- reduce precision in testing
I think I prefer 2, I doubt this is that expensive an operation, maybe I'll do a timing test, although tbh this is me making work for myself!
ah I think I have found it.
indicies = np.argsort(X)
give different results for the same X with and without numba. If the one in a different order is right on the boundary of q1 and q2, they can change bins and slightly change the calculation. Will dump this here so I can find it again.
@njit(fastmath=True, cache=True)
def _SB_MotifThree_quantile_hh_NUMBA(X):
# Shannon entropy of two successive letters in equiprobable 3-letter
# symbolization.
indicies = np.argsort(X)
bins = np.zeros(len(X))
q1 = int(len(X) / 3)
q2 = q1 * 2
print("q1 = ", q1, " q2 = ", q2)
l1 = np.zeros(q1, dtype=np.int_)
for i in range(q1):
l1[i] = indicies[i]
l2 = np.zeros(q1, dtype=np.int_)
c1 = 0
for i in range(q1, q2):
bins[indicies[i]] = 1
l2[c1] = indicies[i]
c1 += 1
l3 = np.zeros(len(indicies) - q2, dtype=np.int_)
c2 = 0
for i in range(q2, len(indicies)):
bins[indicies[i]] = 2
l3[c2] = indicies[i]
c2 += 1
bins1 = bins
found_last = False
nsum = 0
for i in range(3):
if i == 0:
o = l1
elif i == 1:
o = l2
else:
o = l3
if not found_last:
for n in range(len(o)):
if o[n] == len(X) - 1:
o = np.delete(o, n)
break
for n in range(3):
nsum2 = 0.0
for v in o:
if bins[v + 1] == n:
nsum2 += 1
if nsum2 > 0:
nsum2 = nsum2/99.0
nsum += nsum2 * np.log(nsum2)
return -nsum, bins, indicies
# @njit(fastmath=True, cache=True)
def _SB_MotifThree_quantile_hh(X):
# Shannon entropy of two successive letters in equiprobable 3-letter
# symbolization.
indicies = np.argsort(X)
bins = np.zeros(len(X))
q1 = int(len(X) / 3)
q2 = q1 * 2
l1 = np.zeros(q1, dtype=np.int_)
for i in range(q1):
l1[i] = indicies[i]
l2 = np.zeros(q1, dtype=np.int_)
c1 = 0
for i in range(q1, q2):
bins[indicies[i]] = 1
l2[c1] = indicies[i]
c1 += 1
l3 = np.zeros(len(indicies) - q2, dtype=np.int_)
c2 = 0
for i in range(q2, len(indicies)):
bins[indicies[i]] = 2
l3[c2] = indicies[i]
c2 += 1
found_last = False
nsum = 0
for i in range(3):
if i == 0:
o = l1
elif i == 1:
o = l2
else:
o = l3
if not found_last:
for n in range(len(o)):
if o[n] == len(X) - 1:
o = np.delete(o, n)
break
for n in range(3):
nsum2 = 0.0
for v in o:
if bins[v + 1] == n:
nsum2 += 1
if nsum2 > 0:
nsum2 = nsum2/99.0
nsum += nsum2 * np.log(nsum2)
return -nsum, bins, indicies
X_train, _ = load_basic_motions(split="train")
i =2
j=0
x1 = X_train[i][j]
t1, bins1, indx1= _SB_MotifThree_quantile_hh_NUMBA(x1)
t2, bins2, indx2 = _SB_MotifThree_quantile_hh(x1)
print(" i = ", i, " j = ", j)
print(type(t1))
print(type(t2))
print(t1)
print(t2)
print(t1 == t2)
for i in range(len(bins1)):
if(bins1[i] != bins2[i]):
print(i,",",bins1[i],",", bins2[i])
for i in range(len(indx1)):
if(indx1[i] != indx2[i]):
print(i,",",indx1[i],",", indx2[i])
output. 50 and 51 are reversed in the ordering, and that is on the boundary of 65/66
and the two values are the same
print(x1[50], x1[51]) print(x1[36], x1[37])
gives
print(x1[50], x1[51]) print(x1[36], x1[37])
basically if they are equal it seems that numba call puts the lower index first, normal call the higher one. So short of implementing argsort, this does not seem solvable. This is a fast function, but numba does make it about 5x-10x faster.
personally I would prefer to remove numba in the name of determinism, but equally could just leave it as is. Times in seconds, 3 million length takes under a second with numba, around five seconds without
# timing
for i in range(100000,10000000, 100000):
x1 = np.random.rand(i)
t1 = time.time()
_SB_MotifThree_quantile_hh_NUMBA(x1)
t2 = time.time()-t1
t3 = time.time()
_SB_MotifThree_quantile_hh(x1)
t4 = time.time()-t1
print(i,",", t2,",", t4)
simpler example
# @njit(fastmath=True, cache=True)
def argsort_normal(X):
return np.argsort(X)
@njit(fastmath=True, cache=True)
def argsort_numba(X):
return np.argsort(X)
X_train, _ = load_basic_motions(split="train")
i =2
j=0
x1 = X_train[i][j]
indx1 = argsort_normal(x1)
indx2 = argsort_numba(x1)
for i in range(len(indx1)):
if(indx1[i] != indx2[i]):
print(i,", index1 = ",indx1[i]," value1 = ,",x1[indx1[i]],"index2 = ",
indx2[i], " value2 = ",x1[indx2[i]])