[BUG] Race condition in CMake build for generated `H5lib_settings.c`

[haampie]

Describe the bug

In Spack CI pipelines we sometimes see an issue where shared/H5lib_settings.c is seemingly an empty file:

[ 81%] Building C object src/CMakeFiles/hdf5-shared.dir/shared/H5lib_settings.c.o
cd /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/src && /builds/spack/spack/lib/spack/env/gcc/gcc -DH5_BUILT_AS_DYNAMIC_LIB -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200809L -Dhdf5_shared_EXPORTS -I/home/software/spack/[padded-to-384-chars]/linux-amzn2-x86_64_v3/gcc-7.3.1/zlib-1.2.13-ab6htacxtycs5ui3rx76qy2d3f5kyyqz/include -I/tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-src/src -I/tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/src -std=c99 -Wno-error=implicit-function-declaration -fstdarg-opt -fmessage-length=0 -O2 -g -DNDEBUG -fPIC -Wall -Wcast-qual -Wconversion -Wextra -Wfloat-equal -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-include-dirs -Wshadow -Wundef -Wwrite-strings -pedantic -Wno-c++-compat -Wbad-function-cast -Wimplicit-function-declaration -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpointer-sign -Wpointer-to-int-cast -Wint-to-pointer-cast -Wredundant-decls -Wstrict-prototypes -Wswitch -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wunused-but-set-variable -Wformat -finline-functions -Wno-aggregate-return -Wno-inline -Wno-missing-format-attribute -Wno-missing-noreturn -Wno-overlength-strings -Wlarger-than=2560 -Wlogical-op -Wframe-larger-than=16384 -Wpacked-bitfield-compat -Wsync-nand -Wstrict-overflow=5 -Wno-unsuffixed-float-constants -Wdouble-promotion -Wtrampolines -Wstack-usage=8192 -Wmaybe-uninitialized -Wno-jump-misses-init -Wno-suggest-attribute=const -Wno-suggest-attribute=noreturn -Wno-suggest-attribute=pure -Wno-suggest-attribute=format -Wdate-time -Warray-bounds=2 -Wc99-c11-compat -Wincompatible-pointer-types -Wint-conversion -Wduplicated-cond -Whsa -Wnormalized -Wnull-dereference -Wunused-const-variable -Walloca -Walloc-zero -Wduplicated-branches -Wformat-overflow=2 -Wformat-truncation=1 -Wrestrict -std=gnu99 -MD -MT src/CMakeFiles/hdf5-shared.dir/shared/H5lib_settings.c.o -MF CMakeFiles/hdf5-shared.dir/shared/H5lib_settings.c.o.d -o CMakeFiles/hdf5-shared.dir/shared/H5lib_settings.c.o -c /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/src/shared/H5lib_settings.c
/tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/src/shared/H5lib_settings.c:1:0: warning: ISO C forbids an empty translation unit [-Wpedantic]

This causes a build failure later, as the global is missing:

[ 93%] Linking C executable ../../bin/mirror_server_stop
cd /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/utils/mirror_vfd && /home/software/spack/[padded-to-384-chars]/linux-amzn2-x86_64_v3/gcc-7.3.1/cmake-3.24.2-nt5f7rwmajuyvvngu7ismtoo6x6ccwvm/bin/cmake -E cmake_link_script CMakeFiles/mirror_server_stop.dir/link.txt --verbose=1
/builds/spack/spack/lib/spack/env/gcc/gcc   -std=c99 -Wno-error=implicit-function-declaration -fstdarg-opt -fmessage-length=0 -O2 -g -DNDEBUG CMakeFiles/mirror_server_stop.dir/mirror_server_stop.c.o -o ../../bin/mirror_server_stop  -Wl,-rpath,"\$ORIGIN/../lib:\$ORIGIN/" ../../bin/libhdf5.so.200.2.0 -ldl 
../../bin/libhdf5.so.200.2.0: undefined reference to `H5libhdf5_settings'

This is because H5lib_settings.c is generated twice (shared, static) in the same location, and copied into the shared dir.

So, likely what happens: shared generates the file -> static re-generates the file, truncating it -> cmake copies the file into shared -> static finishes generating the file, and we end up with an empty source file.

How about generating H5lib_settings.c once, or generating it twice but immediately in the right directory.

Expected behavior

The shared/H5lib_settings.c file should not be empty

Platform (please complete the following information)

[email protected]%[email protected]~cxx~fortran+hl~ipo~java~mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=RelWithDebInfo arch=linux-amzn2-x86_64_v3
    ^[email protected]%[email protected]~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=linux-amzn2-x86_64_v3
    ^[email protected]%[email protected] build_system=autotools arch=linux-amzn2-x86_64_v3
    ^[email protected]%[email protected]+optimize+pic+shared build_system=makefile arch=linux-amzn2-x86_64_v3

Additional context

The relevant bits of the log:

make  -f src/CMakeFiles/gen_hdf5-static.dir/build.make src/CMakeFiles/gen_hdf5-static.dir/build
make[2]: Entering directory '/tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx'
[  1%] Create H5Tinit.c
cd /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/src && /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/bin/H5detect H5Tinit.c
cd /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/src && /home/software/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/cmake-3.24.2-nt5f7rwmajuyvvngu7ismtoo6x6ccwvm/bin/cmake -E touch gen_SRCS.stamp1
[  1%] Create H5lib_settings.c
cd /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/src && /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/bin/H5make_libsettings H5lib_settings.c
cd /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/src && /home/software/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/cmake-3.24.2-nt5f7rwmajuyvvngu7ismtoo6x6ccwvm/bin/cmake -E touch gen_SRCS.stamp2
[  1%] Generation target files
make[2]: Leaving directory '/tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx'
[  1%] Built target gen_hdf5-static
make  -f src/CMakeFiles/gen_hdf5-shared.dir/build.make src/CMakeFiles/gen_hdf5-shared.dir/depend
make[2]: Entering directory '/tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx'
cd /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx && /home/software/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/cmake-3.24.2-nt5f7rwmajuyvvngu7ismtoo6x6ccwvm/bin/cmake -E cmake_depends "Unix Makefiles" /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-src /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-src/src /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/src /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/src/CMakeFiles/gen_hdf5-shared.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx'
make  -f src/CMakeFiles/gen_hdf5-shared.dir/build.make src/CMakeFiles/gen_hdf5-shared.dir/build
make[2]: Entering directory '/tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx'
[  1%] Create H5lib_settings.c
cd /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/src && /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/bin/H5make_libsettings H5lib_settings.c
cd /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/src && /home/software/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/cmake-3.24.2-nt5f7rwmajuyvvngu7ismtoo6x6ccwvm/bin/cmake -E touch gen_SRCS.stamp2
[  1%] Create H5Tinit.c
cd /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/src && /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/bin/H5detect H5Tinit.c
cd /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/src && /home/software/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/cmake-3.24.2-nt5f7rwmajuyvvngu7ismtoo6x6ccwvm/bin/cmake -E touch gen_SRCS.stamp1
[  1%] Copy H5lib_settings.c to shared folder
cd /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/src && /home/software/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/cmake-3.24.2-nt5f7rwmajuyvvngu7ismtoo6x6ccwvm/bin/cmake -E copy_if_different H5lib_settings.c shared/H5lib_settings.c
cd /tmp/root/spack-stage/spack-stage-hdf5-1.12.2-6w3mfyxp2od5uxq665zvmn6k5tacmwyn/spack-build-6w3mfyx/src && /home/software/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/cmake-3.24.2-nt5f7rwmajuyvvngu7ismtoo6x6ccwvm/bin/cmake -E touch shared/shared_gen_SRCS.stamp2

Full log

[byrnHDF]

Looks like it also generates H5Tinit.c twice. The purpose of the stamps was to prevent the re-execution of the builds. Not sure why CMake is deciding to rebuild the dependencies.

[haampie]

CMake works in mysterious ways

[derobins]

I'm going to go ahead and close this. We've removed H5detect and H5make_libsettings in develop and that will move to 1.14 for the 1.14.2 release.

FWIW, anyone using 1.12 should move to 1.14. We're retiring the 1.12 line as the VOL interface is incomplete.