gaussian_query_test.py tests fail

[kiddyboots216]

Cloned the repo, conda install tensorflow, followed setup instructions inside the repo and then tried to run gaussian_query_test.py but it seems that some of the tests fail.

Running tests under Python 3.7.5: //anaconda3/envs/tf/bin/python
[ RUN      ] GaussianQueryTest.test_gaussian_average_no_noise
2019-11-14 00:57:05.454391: I tensorflow/core/platform/cpu_feature_guard.cc:145] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in non-MKL-DNN operations, rebuild TensorFlow with the appropriate compiler flags.
2019-11-14 00:57:05.456672: I tensorflow/core/common_runtime/process_util.cc:115] Creating new thread pool with default inter op setting: 4. Tune using inter_op_parallelism_threads for best performance.
[       OK ] GaussianQueryTest.test_gaussian_average_no_noise
[ RUN      ] GaussianQueryTest.test_gaussian_average_with_noise
[  FAILED  ] GaussianQueryTest.test_gaussian_average_with_noise
[ RUN      ] GaussianQueryTest.test_gaussian_sum_merge
[       OK ] GaussianQueryTest.test_gaussian_sum_merge
[ RUN      ] GaussianQueryTest.test_gaussian_sum_no_clip_no_noise
[       OK ] GaussianQueryTest.test_gaussian_sum_no_clip_no_noise
[ RUN      ] GaussianQueryTest.test_gaussian_sum_with_changing_clip_no_noise
[  FAILED  ] GaussianQueryTest.test_gaussian_sum_with_changing_clip_no_noise
[ RUN      ] GaussianQueryTest.test_gaussian_sum_with_clip_no_noise
[       OK ] GaussianQueryTest.test_gaussian_sum_with_clip_no_noise
[ RUN      ] GaussianQueryTest.test_gaussian_sum_with_noise
[  FAILED  ] GaussianQueryTest.test_gaussian_sum_with_noise
[ RUN      ] GaussianQueryTest.test_incompatible_records_too_few_on_left
[       OK ] GaussianQueryTest.test_incompatible_records_too_few_on_left
[ RUN      ] GaussianQueryTest.test_incompatible_records_too_few_on_right
[       OK ] GaussianQueryTest.test_incompatible_records_too_few_on_right
[ RUN      ] GaussianQueryTest.test_incompatible_records_type_mismatch
[       OK ] GaussianQueryTest.test_incompatible_records_type_mismatch
[ RUN      ] GaussianQueryTest.test_session
[  SKIPPED ] GaussianQueryTest.test_session
======================================================================
ERROR: test_gaussian_sum_with_changing_clip_no_noise (__main__.GaussianQueryTest)
test_gaussian_sum_with_changing_clip_no_noise (__main__.GaussianQueryTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "gaussian_query_test.py", line 62, in test_gaussian_sum_with_changing_clip_no_noise
    l2_norm_clip_placeholder = tf.compat.v1.placeholder(tf.float32)
  File "//anaconda3/envs/tf/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py", line 2627, in placeholder
    raise RuntimeError("tf.placeholder() is not compatible with "
RuntimeError: tf.placeholder() is not compatible with eager execution.

======================================================================
FAIL: test_gaussian_average_with_noise (__main__.GaussianQueryTest)
test_gaussian_average_with_noise (__main__.GaussianQueryTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "gaussian_query_test.py", line 149, in test_gaussian_average_with_noise
    self.assertNear(result_stddev, avg_stddev, 0.1)
  File "//anaconda3/envs/tf/lib/python3.7/site-packages/tensorflow_core/python/framework/test_util.py", line 1122, in decorated
    return f(*args, **kwds)
  File "//anaconda3/envs/tf/lib/python3.7/site-packages/tensorflow_core/python/framework/test_util.py", line 2225, in assertNear
    (f1, f2, err, " (%s)" % msg if msg is not None else ""))
AssertionError: False is not true : 0.000000 != 0.500000 +/- 0.100000

======================================================================
FAIL: test_gaussian_sum_with_noise (__main__.GaussianQueryTest)
test_gaussian_sum_with_noise (__main__.GaussianQueryTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "gaussian_query_test.py", line 93, in test_gaussian_sum_with_noise
    self.assertNear(result_stddev, stddev, 0.1)
  File "//anaconda3/envs/tf/lib/python3.7/site-packages/tensorflow_core/python/framework/test_util.py", line 1122, in decorated
    return f(*args, **kwds)
  File "//anaconda3/envs/tf/lib/python3.7/site-packages/tensorflow_core/python/framework/test_util.py", line 2225, in assertNear
    (f1, f2, err, " (%s)" % msg if msg is not None else ""))
AssertionError: False is not true : 0.000000 != 1.000000 +/- 0.100000

----------------------------------------------------------------------
Ran 11 tests in 0.645s

FAILED (failures=2, errors=1, skipped=1)

[AndreiBarsan]

The first test is definitely related to TF 2.0 (it enables eager execution by default), which I think the authors mention is in progress of being fixed).

I don't know what could be causing the latter two errors, but it may be some v2 change in default dtypes, causing some rounding issues?

[kiddyboots216]

@AndreiBarsan how could it be a rounding error? The tests show '0 != 1.0' there is no rounding error that would lead to 1 sig fig of difference.

[AndreiBarsan]

Oh, sorry for not being clear. My hypothesis was that it could be something that's now being, e.g., cast to an integer type without rounding, whereas it used to be rounded before (e.g., int(0.5) = 0 but int(round(0.5)) = 1), but I'm not sure.

[kiddyboots216]

In the first test the expected value is 0.5, in the second, 1.0; the code returns 0.0 for both. Your theory is plausible but it would only explain one of these failures, right? (it could be 0.5 and rounded down to 0.0, or it could be 0.5 that wasn't supposed to be rounded)