MOE icon indicating copy to clipboard operation
MOE copied to clipboard

Fix flakey tests, increase tol, p1 or 2 to set threads RNG

Open sc932 opened this issue 10 years ago • 1 comments

 moe.tests.optimal_learning.python.cpp_wrappers.expected_improvement_test ExpectedImprovementTest.test_python_and_cpp_return_same_1d_analytic_ei_and_gradient
Traceback (most recent call last):
  File "./moe/tests/optimal_learning/python/cpp_wrappers/expected_improvement_test.py", line 87, in test_python_and_cpp_return_same_1d_analytic_ei_and_gradient
    self.assert_vector_within_relative(python_grad_ei, cpp_grad_ei, grad_ei_tolerance)
  File "./moe/tests/optimal_learning/python/optimal_learning_test_case.py", line 62, in assert_vector_within_relative
    self.assert_scalar_within_relative(value[index], truth[index], tol)
  File "./moe/tests/optimal_learning/python/optimal_learning_test_case.py", line 41, in assert_scalar_within_relative
    message='value = {0:.18E}, truth = {1:.18E}, diff = {2:.18E}, tol = {3:.18E}'.format(value, truth, diff, tol),
AssertionError: value = -1.710635215382066629E-05, truth = -1.710635215381947367E-05, diff = 6.971810114804449073E-14, tol = 5.999999999999999677E-14

FAIL in 0.38s

AND

moe.
tests.optimal_learning.python.cpp_wrappers.gaussian_process_test GaussianProcessTest.test_python_and_cpp_return_same_cholesky_variance_and_gradient
Traceback (most recent call last):
  File "./moe/tests/optimal_learning/python/cpp_wrappers/gaussian_process_test.py", line 132, in test_python_and_cpp_return_same_cholesky_variance_and_gradient
    self.assert_vector_within_relative(python_grad_var, cpp_grad_var, grad_var_tolerance)
  File "./moe/tests/optimal_learning/python/optimal_learning_test_case.py", line 62, in assert_vector_within_relative
    self.assert_scalar_within_relative(value[index], truth[index], tol)
  File "./moe/tests/optimal_learning/python/optimal_learning_test_case.py", line 41, in assert_scalar_within_relative
    message='value = {0:.18E}, truth = {1:.18E}, diff = {2:.18E}, tol = {3:.18E}'.format(value, truth, diff, tol),
AssertionError: value = -1.810166738444277133E-06, truth = -1.810166738419891690E-06, diff = 1.347137960231614092E-11, tol = 3.000000000000000142E-12

FAIL in 1.38s

sc932 avatar Jun 25 '14 18:06 sc932

tolerances decreased so these tests pass. I'm bumping the priority of this ticket down. It is now about making these tests more repeatable (i.e., by setting RNG)

suntzu86 avatar Jun 25 '14 21:06 suntzu86