darktable icon indicating copy to clipboard operation
darktable copied to clipboard
verified publisher icon darktable-org

march/mtune tests failure

Open kmilos opened this issue 1 year ago • 5 comments

Describe the bug

Both march and mtune test fail when building w/ either GCC 13 or Clang 17/18 under MSYS2, with either BINARY_PACKAGE_BUILD option.

  -- Performing Test MTUNEGENERIC
  -- Performing Test MTUNEGENERIC - Failed
  CMake Warning at cmake/march-mtune.cmake:30 (message):
    Do not know which -march/-mtune to pass! Resulting binaries may be broken!
  Call Stack (most recent call first):
    src/CMakeLists.txt:638 (include)

As a result, loop vectorization seems to be dropped.

Interestingly, the equivalent test passes earlier for the RawSpeed module:

  -- Checking for -mtune=generic support
  -- Performing Test RAWSPEED_MTUNEGENERIC
  -- Performing Test RAWSPEED_MTUNEGENERIC - Success
  -- Checking for -mtune=generic support - works

The following standalone CMake test also works:

cmake_minimum_required(VERSION 3.18)

project(test C)

include(CheckCCompilerFlag)
check_c_compiler_flag("-march=native" MARCHNATIVE)

Steps to reproduce

Build under MSYS2 UCRT64 or CLANG64 environment.

Expected behavior

No response

Logfile | Screenshot | Screencast

No response

Commit

No response

Where did you obtain darktable from?

self compiled

darktable version

4.7.0+784~gf983fb2268

What OS are you using?

Windows

What is the version of your OS?

Windows 11 23H2

Describe your system?

No response

Are you using OpenCL GPU in darktable?

None

If yes, what is the GPU card and driver?

No response

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

No response

kmilos avatar Mar 18 '24 15:03 kmilos

OK, Houston, we have a problem. Examining the nightly build logs showed that this problem appeared on github runners on February 15th. Before that, everything was fine. Either some questionable changes were made to our cmake files the day before, or there was some buggy update in the msys2 packages.

victoryforce avatar Mar 20 '24 14:03 victoryforce

The log of exactly what failed is in build\CMakeFiles\CMakeConfigureLog.yaml.

In my local build it contains the following (hidden behind the header to not pollute the comment).

Details 

kind: "try_compile-v1"
backtrace:
  - "C:/msys64/ucrt64/share/cmake/Modules/Internal/CheckSourceCompiles.cmake:101 (try_compile)"
  - "C:/msys64/ucrt64/share/cmake/Modules/Internal/CheckCompilerFlag.cmake:18 (cmake_check_source_compiles)"
  - "C:/msys64/ucrt64/share/cmake/Modules/CheckCCompilerFlag.cmake:51 (cmake_check_compiler_flag)"
  - "cmake/march-mtune.cmake:5 (CHECK_C_COMPILER_FLAG)"
  - "src/CMakeLists.txt:647 (include)"
checks:
  - "Performing Test MARCHNATIVE"
directories:
  source: "C:/msys64/home/Victor/darktable/build/CMakeFiles/CMakeScratch/TryCompile-2yzent"
  binary: "C:/msys64/home/Victor/darktable/build/CMakeFiles/CMakeScratch/TryCompile-2yzent"
cmakeVariables:
  CMAKE_C_FLAGS: " -Wall -Wno-format -Wshadow -Wtype-limits -Wvla -Wold-style-declaration -Wmaybe-uninitialized -Wno-unknown-pragmas -Wno-error=varargs -Wno-format-truncation -Wno-error=address-of-packed-member -fopenmp"
  CMAKE_C_FLAGS_DEBUG: "-g"
  CMAKE_EXE_LINKER_FLAGS: " -Wl,--enable-runtime-pseudo-reloc -municode"
  CMAKE_MODULE_PATH: "C:/msys64/home/Victor/darktable/cmake/modules/;C:/msys64/home/Victor/darktable/cmake/"
  CMAKE_POSITION_INDEPENDENT_CODE: "ON"
buildResult:
  variable: "MARCHNATIVE"
  cached: true
  stdout: |
    Change Dir: 'C:/msys64/home/Victor/darktable/build/CMakeFiles/CMakeScratch/TryCompile-2yzent'
    
    Run Build Command(s): C:/msys64/ucrt64/bin/ninja.exe -v cmTC_8793d
    [1/2] C:\\msys64\\ucrt64\\bin\\cc.exe -DMARCHNATIVE  -Wall -Wno-format -Wshadow -Wtype-limits -Wvla -Wold-style-declaration -Wmaybe-uninitialized -Wno-unknown-pragmas -Wno-error=varargs -Wno-format-truncation -Wno-error=address-of-packed-member -fopenmp  -std=c99   -march=native -o CMakeFiles/cmTC_8793d.dir/src.c.obj -c C:/msys64/home/Victor/darktable/build/CMakeFiles/CMakeScratch/TryCompile-2yzent/src.c
    [2/2] C:\\WINDOWS\\system32\\cmd.exe /C "cd . && C:\\msys64\\ucrt64\\bin\\cc.exe -Wall -Wno-format -Wshadow -Wtype-limits -Wvla -Wold-style-declaration -Wmaybe-uninitialized -Wno-unknown-pragmas -Wno-error=varargs -Wno-format-truncation -Wno-error=address-of-packed-member -fopenmp -Wl,--enable-runtime-pseudo-reloc -municode CMakeFiles/cmTC_8793d.dir/src.c.obj -o cmTC_8793d.exe -Wl,--out-implib,libcmTC_8793d.dll.a -Wl,--major-image-version,0,--minor-image-version,0  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."
    FAILED: cmTC_8793d.exe 
    C:\\WINDOWS\\system32\\cmd.exe /C "cd . && C:\\msys64\\ucrt64\\bin\\cc.exe -Wall -Wno-format -Wshadow -Wtype-limits -Wvla -Wold-style-declaration -Wmaybe-uninitialized -Wno-unknown-pragmas -Wno-error=varargs -Wno-format-truncation -Wno-error=address-of-packed-member -fopenmp -Wl,--enable-runtime-pseudo-reloc -municode CMakeFiles/cmTC_8793d.dir/src.c.obj -o cmTC_8793d.exe -Wl,--out-implib,libcmTC_8793d.dll.a -Wl,--major-image-version,0,--minor-image-version,0  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."
    C:/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../lib/libmingw32.a(lib64_libmingw32_a-ucrtexewin.o): in function `wmain':\x0d
    C:/M/B/src/mingw-w64/mingw-w64-crt/crt/ucrtexewin.c:67:(.text+0xa5): undefined reference to `wWinMain'\x0d
    collect2.exe: error: ld returned 1 exit status
    ninja: build stopped: subcommand failed.
    
  exitCode: 1

kind: "try_compile-v1"
backtrace:
  - "C:/msys64/ucrt64/share/cmake/Modules/Internal/CheckSourceCompiles.cmake:101 (try_compile)"
  - "C:/msys64/ucrt64/share/cmake/Modules/Internal/CheckCompilerFlag.cmake:18 (cmake_check_source_compiles)"
  - "C:/msys64/ucrt64/share/cmake/Modules/CheckCCompilerFlag.cmake:51 (cmake_check_compiler_flag)"
  - "cmake/march-mtune.cmake:11 (CHECK_C_COMPILER_FLAG)"
  - "src/CMakeLists.txt:647 (include)"
checks:
  - "Performing Test MTUNENATIVE"
directories:
  source: "C:/msys64/home/Victor/darktable/build/CMakeFiles/CMakeScratch/TryCompile-omjaen"
  binary: "C:/msys64/home/Victor/darktable/build/CMakeFiles/CMakeScratch/TryCompile-omjaen"
cmakeVariables:
  CMAKE_C_FLAGS: " -Wall -Wno-format -Wshadow -Wtype-limits -Wvla -Wold-style-declaration -Wmaybe-uninitialized -Wno-unknown-pragmas -Wno-error=varargs -Wno-format-truncation -Wno-error=address-of-packed-member -fopenmp"
  CMAKE_C_FLAGS_DEBUG: "-g"
  CMAKE_EXE_LINKER_FLAGS: " -Wl,--enable-runtime-pseudo-reloc -municode"
  CMAKE_MODULE_PATH: "C:/msys64/home/Victor/darktable/cmake/modules/;C:/msys64/home/Victor/darktable/cmake/"
  CMAKE_POSITION_INDEPENDENT_CODE: "ON"
buildResult:
  variable: "MTUNENATIVE"
  cached: true
  stdout: |
    Change Dir: 'C:/msys64/home/Victor/darktable/build/CMakeFiles/CMakeScratch/TryCompile-omjaen'
    
    Run Build Command(s): C:/msys64/ucrt64/bin/ninja.exe -v cmTC_74bef
    [1/2] C:\\msys64\\ucrt64\\bin\\cc.exe -DMTUNENATIVE  -Wall -Wno-format -Wshadow -Wtype-limits -Wvla -Wold-style-declaration -Wmaybe-uninitialized -Wno-unknown-pragmas -Wno-error=varargs -Wno-format-truncation -Wno-error=address-of-packed-member -fopenmp  -std=c99   -mtune=native -o CMakeFiles/cmTC_74bef.dir/src.c.obj -c C:/msys64/home/Victor/darktable/build/CMakeFiles/CMakeScratch/TryCompile-omjaen/src.c
    [2/2] C:\\WINDOWS\\system32\\cmd.exe /C "cd . && C:\\msys64\\ucrt64\\bin\\cc.exe -Wall -Wno-format -Wshadow -Wtype-limits -Wvla -Wold-style-declaration -Wmaybe-uninitialized -Wno-unknown-pragmas -Wno-error=varargs -Wno-format-truncation -Wno-error=address-of-packed-member -fopenmp -Wl,--enable-runtime-pseudo-reloc -municode CMakeFiles/cmTC_74bef.dir/src.c.obj -o cmTC_74bef.exe -Wl,--out-implib,libcmTC_74bef.dll.a -Wl,--major-image-version,0,--minor-image-version,0  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."
    FAILED: cmTC_74bef.exe 
    C:\\WINDOWS\\system32\\cmd.exe /C "cd . && C:\\msys64\\ucrt64\\bin\\cc.exe -Wall -Wno-format -Wshadow -Wtype-limits -Wvla -Wold-style-declaration -Wmaybe-uninitialized -Wno-unknown-pragmas -Wno-error=varargs -Wno-format-truncation -Wno-error=address-of-packed-member -fopenmp -Wl,--enable-runtime-pseudo-reloc -municode CMakeFiles/cmTC_74bef.dir/src.c.obj -o cmTC_74bef.exe -Wl,--out-implib,libcmTC_74bef.dll.a -Wl,--major-image-version,0,--minor-image-version,0  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."
    C:/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../lib/libmingw32.a(lib64_libmingw32_a-ucrtexewin.o): in function `wmain':\x0d
    C:/M/B/src/mingw-w64/mingw-w64-crt/crt/ucrtexewin.c:67:(.text+0xa5): undefined reference to `wWinMain'\x0d
    collect2.exe: error: ld returned 1 exit status
    ninja: build stopped: subcommand failed.
    
  exitCode: 1

kind: "try_compile-v1"
backtrace:
  - "C:/msys64/ucrt64/share/cmake/Modules/Internal/CheckSourceCompiles.cmake:101 (try_compile)"
  - "C:/msys64/ucrt64/share/cmake/Modules/Internal/CheckCompilerFlag.cmake:18 (cmake_check_source_compiles)"
  - "C:/msys64/ucrt64/share/cmake/Modules/CheckCCompilerFlag.cmake:51 (cmake_check_compiler_flag)"
  - "cmake/march-mtune.cmake:16 (CHECK_C_COMPILER_FLAG)"
  - "src/CMakeLists.txt:647 (include)"
checks:
  - "Performing Test MTUNEGENERIC"
directories:
  source: "C:/msys64/home/Victor/darktable/build/CMakeFiles/CMakeScratch/TryCompile-8dgh6j"
  binary: "C:/msys64/home/Victor/darktable/build/CMakeFiles/CMakeScratch/TryCompile-8dgh6j"
cmakeVariables:
  CMAKE_C_FLAGS: " -Wall -Wno-format -Wshadow -Wtype-limits -Wvla -Wold-style-declaration -Wmaybe-uninitialized -Wno-unknown-pragmas -Wno-error=varargs -Wno-format-truncation -Wno-error=address-of-packed-member -fopenmp"
  CMAKE_C_FLAGS_DEBUG: "-g"
  CMAKE_EXE_LINKER_FLAGS: " -Wl,--enable-runtime-pseudo-reloc -municode"
  CMAKE_MODULE_PATH: "C:/msys64/home/Victor/darktable/cmake/modules/;C:/msys64/home/Victor/darktable/cmake/"
  CMAKE_POSITION_INDEPENDENT_CODE: "ON"
buildResult:
  variable: "MTUNEGENERIC"
  cached: true
  stdout: |
    Change Dir: 'C:/msys64/home/Victor/darktable/build/CMakeFiles/CMakeScratch/TryCompile-8dgh6j'
    
    Run Build Command(s): C:/msys64/ucrt64/bin/ninja.exe -v cmTC_68890
    [1/2] C:\\msys64\\ucrt64\\bin\\cc.exe -DMTUNEGENERIC  -Wall -Wno-format -Wshadow -Wtype-limits -Wvla -Wold-style-declaration -Wmaybe-uninitialized -Wno-unknown-pragmas -Wno-error=varargs -Wno-format-truncation -Wno-error=address-of-packed-member -fopenmp  -std=c99   -mtune=generic -o CMakeFiles/cmTC_68890.dir/src.c.obj -c C:/msys64/home/Victor/darktable/build/CMakeFiles/CMakeScratch/TryCompile-8dgh6j/src.c
    [2/2] C:\\WINDOWS\\system32\\cmd.exe /C "cd . && C:\\msys64\\ucrt64\\bin\\cc.exe -Wall -Wno-format -Wshadow -Wtype-limits -Wvla -Wold-style-declaration -Wmaybe-uninitialized -Wno-unknown-pragmas -Wno-error=varargs -Wno-format-truncation -Wno-error=address-of-packed-member -fopenmp -Wl,--enable-runtime-pseudo-reloc -municode CMakeFiles/cmTC_68890.dir/src.c.obj -o cmTC_68890.exe -Wl,--out-implib,libcmTC_68890.dll.a -Wl,--major-image-version,0,--minor-image-version,0  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."
    FAILED: cmTC_68890.exe 
    C:\\WINDOWS\\system32\\cmd.exe /C "cd . && C:\\msys64\\ucrt64\\bin\\cc.exe -Wall -Wno-format -Wshadow -Wtype-limits -Wvla -Wold-style-declaration -Wmaybe-uninitialized -Wno-unknown-pragmas -Wno-error=varargs -Wno-format-truncation -Wno-error=address-of-packed-member -fopenmp -Wl,--enable-runtime-pseudo-reloc -municode CMakeFiles/cmTC_68890.dir/src.c.obj -o cmTC_68890.exe -Wl,--out-implib,libcmTC_68890.dll.a -Wl,--major-image-version,0,--minor-image-version,0  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."
    C:/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../lib/libmingw32.a(lib64_libmingw32_a-ucrtexewin.o): in function `wmain':\x0d
    C:/M/B/src/mingw-w64/mingw-w64-crt/crt/ucrtexewin.c:67:(.text+0xa5): undefined reference to `wWinMain'\x0d
    collect2.exe: error: ld returned 1 exit status
    ninja: build stopped: subcommand failed.
    
  exitCode: 1

I don't understand what could cause this undefined reference.

victoryforce avatar Mar 20 '24 15:03 victoryforce

I don't understand what could cause this undefined reference.

Ah, good analysis. I think it's the fact the -municode switch injects this wWinMain() expectation we take care w/ src/win/main_wrapper.h. Explains why the standalone (and RawSpeed) detection works.

Don't know what has changed in the MSYS2 toolchain behaviour though as this wasn't a problem before?

So we have a couple of choices:

  1. Move detection before adding -municode to compiler switches
  2. Try to engage w/ MSYS2 and figure out exactly what has changed
  3. Drop -municode (will require a minimum Windows version bump); I kinda already started this because of the exiv2 issue...

kmilos avatar Mar 20 '24 15:03 kmilos

It's the fact the -municode switch injects this WinMain() expectation we take care w/ src/win/main_wrapper.h.

Yes, that's clear. I also don't understand why this suddenly became a problem at some point.

Explains why the standalone (and RawSpeed) detection works.

Because of the -mconsole default.

victoryforce avatar Mar 20 '24 16:03 victoryforce

  • Move detection before adding -municode to compiler switches

I tested this. It works. I'll make a PR with this fix soon.

  • Try to engage w/ MSYS2 and figure out exactly what has changed

This would make sense, since it is possible that the change made was not sufficiently justified and correct and could be reversed. However, this will definitely take time and if this change is not reversed, we will not get any closer to a solution.

Instead, moving the detection code to a place where it works is a simple and guaranteed solution.

It's an invasive solution that has a big impact on darktable, requires extensive testing, and I wouldn't use it to solve this simple issue.

victoryforce avatar Mar 20 '24 18:03 victoryforce