Are all tests expected to pass?

[norbertwenzel]

I've simply cloned the repo (master & v1.3.6) and used CMake to build and run the tests. But one test fails on consistently on all platforms (Linux, Mac, Windows). I could not find any open issues about a failing test therefore I thought I'd ask whether this is known and/or to be expected.

Steps to reproduce:

Clone repository and change to project's root folder. In this folder run:

cmake -H. -Bbuild
cmake --build build -- -j
cmake -E env CTEST_OUTPUT_ON_FAILURE=1 cmake --build build --target test

Result:

96% tests passed, 1 tests failed out of 24

Total Test time (real) =   0.54 sec

The following tests FAILED:
	 16 - tlmdifc (Failed)
Errors while running CTest
Makefile:108: recipe for target 'test' failed
make: *** [test] Error 8

The test output written to build/examples/tlmdifc.out is:

 final l2 norm of the residuals     0.09063596

 number of function evaluations        21

 exit parameter                         1

 final approximate solution

     0.08241058       1.133037       2.343695
 covariance

   0.0001531202    0.002869942   -0.002656662
    0.002869942     0.09480937    -0.09098997
   -0.002656662    -0.09098997     0.08778729

These results are from a run on Ubuntu Bionic amd64 but the very same test also failed for me on Mac and Windows.

[devernay]

Take a look at the difference with the reference file https://github.com/devernay/cminpack/blob/master/examples/ref/tlmdifc.ref . In this case, the second and third column of the cov matrix are slightly different. I've seen floating point computation, especially when optimized, behave slightly differently on different OSs/architectures. If the difference is non-significative, then you should consider that the tests pass. I agree that the tests should be rewritten to handle that, but "diff" was all I had when the tests were written.

[StefanBruens]

For other architectures (aarch64, ppc64le, ...) every second test fails due to numeric differences.

[devernay]

Yes, this is a known issue, feel free to contribute a better est validation that using diff on the text output