sporadic test failures

[akamaus]

Some tests seem to fail from time to time. Running test suite 100 times in a loop resulted in 29 failures.

File "test_kde.py", line 56, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[0], self.hp_kde_full.bandwidths[0], delta=2e-3)
AssertionError: 0.04618833879866881 != 0.02237181160117373 within 0.002 delta
--
  File "test_kde.py", line 138, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[d], self.hp_kde_full.bandwidths[d], delta=5e-2)
AssertionError: 0.7197090445659962 != 0.8075606853605798 within 0.05 delta
--
  File "test_kde.py", line 56, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[0], self.hp_kde_full.bandwidths[0], delta=2e-3)
AssertionError: 0.31178329664604787 != 0.322800611255376 within 0.002 delta
--
  File "test_kde.py", line 138, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[d], self.hp_kde_full.bandwidths[d], delta=5e-2)
AssertionError: 0.4535001745647519 != 0.5052498165606699 within 0.05 delta
--
  File "test_kde.py", line 56, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[0], self.hp_kde_full.bandwidths[0], delta=2e-3)
AssertionError: 0.9454515348826193 != 0.9392693580115746 within 0.002 delta
--
  File "test_kde.py", line 56, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[0], self.hp_kde_full.bandwidths[0], delta=2e-3)
AssertionError: 0.9456782206454661 != 0.9395671936311828 within 0.002 delta
--
  File "test_kde.py", line 138, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[d], self.hp_kde_full.bandwidths[d], delta=5e-2)
AssertionError: 0.48232806419410734 != 0.5840429875163532 within 0.05 delta
--
  File "test_kde.py", line 138, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[d], self.hp_kde_full.bandwidths[d], delta=5e-2)
AssertionError: 0.7477113291187305 != 0.6346358437160987 within 0.05 delta
--
  File "test_kde.py", line 138, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[d], self.hp_kde_full.bandwidths[d], delta=5e-2)
AssertionError: 0.8583091576429538 != 0.45815843050124433 within 0.05 delta
--
  File "test_kde.py", line 138, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[d], self.hp_kde_full.bandwidths[d], delta=5e-2)
AssertionError: 0.5670394840846824 != 0.671034866569728 within 0.05 delta
--
  File "test_kde.py", line 138, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[d], self.hp_kde_full.bandwidths[d], delta=5e-2)
AssertionError: 0.6716586627109459 != 0.8018535493921695 within 0.05 delta
--
  File "test_kde.py", line 56, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[0], self.hp_kde_full.bandwidths[0], delta=2e-3)
AssertionError: 0.8673847294651753 != 0.8698048648046033 within 0.002 delta
--
  File "test_kde.py", line 56, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[0], self.hp_kde_full.bandwidths[0], delta=2e-3)
AssertionError: 0.02903496614585073 != 0.01370083621999389 within 0.002 delta
--
  File "test_kde.py", line 56, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[0], self.hp_kde_full.bandwidths[0], delta=2e-3)
AssertionError: 0.9454515348826193 != 0.9392661921656029 within 0.002 delta
--
  File "test_kde.py", line 56, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[0], self.hp_kde_full.bandwidths[0], delta=2e-3)
AssertionError: 0.4818229179852173 != 0.4779503016808494 within 0.002 delta
--
  File "test_kde.py", line 138, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[d], self.hp_kde_full.bandwidths[d], delta=5e-2)
AssertionError: 0.16742765643102592 != 0.04232944931290015 within 0.05 delta
--
  File "test_kde.py", line 56, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[0], self.hp_kde_full.bandwidths[0], delta=2e-3)
AssertionError: 0.04782158321091984 != 0.015909283029143306 within 0.002 delta
--
  File "test_kde.py", line 56, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[0], self.hp_kde_full.bandwidths[0], delta=2e-3)
AssertionError: 0.7681037710087533 != 0.764247084344105 within 0.002 delta
--
  File "test_kde.py", line 138, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[d], self.hp_kde_full.bandwidths[d], delta=5e-2)
AssertionError: 0.7119693664556175 != 0.6480948411529247 within 0.05 delta
--
  File "test_kde.py", line 56, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[0], self.hp_kde_full.bandwidths[0], delta=2e-3)
AssertionError: 0.03594243113107432 != 0.01755869747711552 within 0.002 delta
--
  File "test_kde.py", line 138, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[d], self.hp_kde_full.bandwidths[d], delta=5e-2)
AssertionError: 0.8659473488844843 != 0.7647468119143158 within 0.05 delta
--
  File "test_kde.py", line 138, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[d], self.hp_kde_full.bandwidths[d], delta=5e-2)
AssertionError: 0.8355435118154853 != 0.7670194994489465 within 0.05 delta
--
  File "test_kde.py", line 56, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[0], self.hp_kde_full.bandwidths[0], delta=2e-3)
AssertionError: 0.31178329664604787 != 0.322800611255376 within 0.002 delta
--
  File "test_kde.py", line 138, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[d], self.hp_kde_full.bandwidths[d], delta=5e-2)
AssertionError: 0.6910363012632923 != 0.7735924043200492 within 0.05 delta
--
  File "test_kde.py", line 56, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[0], self.hp_kde_full.bandwidths[0], delta=2e-3)
AssertionError: 0.35864261907085565 != 0.3467105134963595 within 0.002 delta
--
  File "test_kde.py", line 138, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[d], self.hp_kde_full.bandwidths[d], delta=5e-2)
AssertionError: 0.7412952221617459 != 0.8044927743114529 within 0.05 delta
--
  File "test_kde.py", line 56, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[0], self.hp_kde_full.bandwidths[0], delta=2e-3)
AssertionError: 0.7677941199500147 != 0.7645337096977642 within 0.002 delta
--
  File "test_kde.py", line 138, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[d], self.hp_kde_full.bandwidths[d], delta=5e-2)
AssertionError: 0.0653788261754362 != 0.12545497668012273 within 0.05 delta
--
  File "test_kde.py", line 56, in test_bandwidths_estimation
    self.assertAlmostEqual(self.sm_kde.bw[0], self.hp_kde_full.bandwidths[0], delta=2e-3)
AssertionError: 0.5517231143430665 != 0.5542477840534359 within 0.002 delta

[sfalkner]

Good catch. The test rely on randomly generated data that is then fit by different kernel density estimators. The particular test failing here is the one for optimizing the bandwidth of said KDEs. Because the different implementations use different optimizers there can be a discrepancy in the resulting 'best bandwidth'. The thresholds in the code are apparently too tight in ~30% of the cases. I shall adjust them to make the tests not fail because of that. Or maybe I will test for this condition, and if it is not fulfilled, more points will be added, which makes these values more stable. Thanks again for pointing this out!