Catch's ApprovalTests test fail if build directory was reached via a symbolic link

[djerius]

Describe the bug When building/testing Catch2 in a directory that was reached via a symbolic link, the ApprovalTests xml.sw test fails.

Here's an example failure line from build/Testing/Temporary/LastTest.log which illustrates the gist of the problem:

+      <Expression success="true" type="REQUIRE" filename="/home/dj/tmp/Catch2-2.9.1/projects/<exe-name>/UsageTests/Decomposition.tests.cpp" >
-      <Expression success="true" type="REQUIRE" filename="projects/<exe-name>/UsageTests/Decomposition.tests.cpp" >

Notice that the directory path has not been correctly sanitized, i.e., it still contains the /home/dj/tmp/Catch2-2.9.1 prefix.

In this case, /home/dj/tmp is a symbolic link to the fully resolved directory name, /data/macabretmp/dj/. The scripts/approvalTests.py program, which is responsible for this test, uses this code in scripts/scriptCommon.py to determine the prefix to strip:

catchPath = os.path.dirname(os.path.realpath( os.path.dirname(sys.argv[0])))

which in my case is equivalent to setting it to /data/macabretmp/dj, not to the required value of /home/dj/tmp.

One approach to avoiding this would be for scripts/scriptCommon.py to parse the directory from one of the CMake artifacts in the build directory.

Expected behavior

I expect that the tests work regardless of whether or not the directory is reached via a symbolic link.

Platform information:

• OS: Debian 9
• Compiler+version: GCC v6.3.0
• Catch version: v2.9.1

[horenmar]

The approval tests no longer resolve symbolic links when obtaining the expected path for sanitization, that should be a sufficient fix for this issue.

There is a potential followup question whether other scripts actually do need to resolve symbolic links for their use, but at a quick glance, it should not matter -- they do not compare the path with anything, but only open files and such.

[djerius]

Thanks!

[horenmar]

I ended up having to revert the fix, as it broke something on our Windows CI (and only CI, it works locally), so until I can figure it out, it can't go into master.

[discapes]

Also getting errors with catch2's ApprovalTests tests for example:

-##teamcity[testFailed name='thrown std::strings are translated' message='Exception.tests.cpp:|n..............................................................> +##teamcity[testFailed name='thrown std::strings are translated' message='/dev/shm/nixroot/BUILDS/NIX-2032523-687298376/SOUrce/Exception.tests.cpp:|n........>