msvc-wine CMake+Ninja: dependencies of RC files missing

resource.h: empty file
embed.rc: #include "resource.h"
main.cpp: int main() {}
CMakeLists.txt:

cmake_minimum_required(VERSION 3.25)
project(rctest)
add_executable(rctest main.cpp embed.rc)

On Windows, touching resource.h and execute ninja -d explain -d keepdepfile -v -j1, RC file is correctly rebuilt.

ninja explain: output CMakeFiles/rctest.dir/embed.rc.res older than most recent input C:/Users/huangqinjin/Projects/rctest/resource.h (7031986049877757 vs 7031986466060779)
ninja explain: CMakeFiles/rctest.dir/embed.rc.res is dirty
ninja explain: rctest.exe is dirty
[1/2] C:/PROGRA~1/CMake/bin/cmcldeps.exe RC C:\Users\huangqinjin\Projects\rctest\embed.rc CMakeFiles\rctest.dir\embed.rc.res.d CMakeFiles\rctest.dir\embed.rc.res "Note: including file: " "C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.35.32215/bin/Hostx64/x64/cl.exe" C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe   -DWIN32 -D_DEBUG /fo CMakeFiles\rctest.dir\embed.rc.res C:\Users\huangqinjin\Projects\rctest\embed.rc
[2/2] cmd.exe /C "cd . && "C:\Program Files\CMake\bin\cmake.exe" -E vs_link_exe --intdir=CMakeFiles\rctest.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MIB055~1\2022\ENTERP~1\VC\Tools\MSVC\1435~1.322\bin\Hostx64\x64\link.exe /nologo CMakeFiles\rctest.dir\main.cpp.obj CMakeFiles\rctest.dir\embed.rc.res  /out:rctest.exe /implib:rctest.lib /pdb:rctest.pdb /version:0.0 /machine:x64 /debug /INCREMENTAL /subsystem:console  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."

cmcldeps.exe generates depfile for embed.rc:

CMakeFiles\\rctest.dir\\embed.rc.res: \
C:\\Users\\huangqinjin\\Projects\\rctest\\resource.h \

But using msvc-wine on Linux, no depfile for embed.rc is generated. It is due to CMAKE_NINJA_CMCLDEPS_RC is only defined on Windows. https://gitlab.kitware.com/cmake/cmake/-/blob/0b552eb877b887638e8130bb6c982106a76827d8/Modules/Platform/Windows-MSVC.cmake#L511

Also see related https://github.com/mstorsjo/msvc-wine/issues/22.

Apr 14 '23 18:04 huangqinjin

I wrote a bash script cmcldeps for the same purpose of cmcldeps.exe on Windows (shamelessly copied the implementation from https://gitlab.kitware.com/cmake/cmake/-/blob/v3.26.1/Source/cmcldeps.cxx). I put the source code here if someone also need it.

This script must be put next to the cmake executable, see https://gitlab.kitware.com/cmake/cmake/-/blob/v3.26.1/Source/cmSystemTools.cxx#L2611-2615.

Edit: There is a loophole invoking "cmcldeps LANG": cmake doesn't check the emptiness of the path, see https://gitlab.kitware.com/cmake/cmake/-/blob/v3.26.1/Source/cmNinjaTargetGenerator.cxx#L804-821. So Ninja actually treats LANG as the executable in the case cmake doesn't find cmcldeps. One can rename this script to LANG (or using symlink) and put it in PATH, if putting the script next to the cmake executable is not doable.

CMakePresets.json

This CMakePresets.json sets two more cmake variables compared to https://github.com/mstorsjo/msvc-wine/issues/62#issue-1667681085. The CMake minimum version could be lowered down.

CMAKE_NINJA_CMCLDEPS_RC=ON: It only sets to ON on Windows since CMake 3.23. https://gitlab.kitware.com/cmake/cmake/-/commit/d49e168e1b32ac7d3ee0a4a52791d28a52d8f615
CMAKE_CL_SHOWINCLUDES_PREFIX="Note: including file: ": Currently CMake successfully extracts the prefix only if paths begin with X:\ or ./ https://gitlab.kitware.com/cmake/cmake/-/blob/fae6e8c2cdb5ce6049439f4defd1367b507d1e4b/Modules/CMakeDetermineCompilerId.cmake#L1147

Edit: Set "Note: including file: " as the default prefix (for English MSVC) in cmcldeps script, which is same as Ninja https://github.com/ninja-build/ninja/blob/v1.11.1/src/clparser.cc#L46, so now no need to set CMAKE_CL_SHOWINCLUDES_PREFIX.

Edit 2: This is actually a regression in CMake 3.26.0 comfirmed by upstream https://gitlab.kitware.com/cmake/cmake/-/issues/24908. It will be fixed in 3.26.4, after that we do not need touch CMAKE_CL_SHOWINCLUDES_PREFIX.

{
  "version": 3,
  "cmakeMinimumRequired": {
    "major": 3,
    "minor": 23,
    "patch": 0
  },
  "configurePresets": [
    {
      "name": "msvc-wine",
      "generator": "Ninja",
      "cacheVariables": {
        "CMAKE_C_COMPILER": "/opt/msvc/bin/x64/cl.exe",
        "CMAKE_CXX_COMPILER": "/opt/msvc/bin/x64/cl.exe",
        "CMAKE_RC_COMPILER": "/opt/msvc/bin/x64/rc.exe",
        "CMAKE_NINJA_CMCLDEPS_RC": "ON",
        "CMAKE_SYSTEM_NAME": "Windows",
        "CMAKE_SYSTEM_VERSION": "10",
        "CMAKE_CROSSCOMPILING_EMULATOR": "env;WINEDEBUG=-all;wine64",
        "CMAKE_BUILD_TYPE": "RelWithDebInfo"
      },
      "condition": {
        "string": "${hostSystemName}",
        "type": "notInList",
        "list" : [ "Windows" ]
      }
    }
  ]
}

cmcldeps

#!/bin/bash

# This is a bash implementation of https://gitlab.kitware.com/cmake/cmake/-/blob/v3.26.1/Source/cmcldeps.cxx

# This script must be put next to the cmake executable, see
# https://gitlab.kitware.com/cmake/cmake/-/blob/v3.26.1/Source/cmSystemTools.cxx#L2611-2615.

# There is a loophole invoking "cmcldeps LANG": cmake doesn't check the emptiness of the path, see
# https://gitlab.kitware.com/cmake/cmake/-/blob/v3.26.1/Source/cmNinjaTargetGenerator.cxx#L804-821.
# So Ninja actually treats LANG as the executable in the case cmake doesn't find cmcldeps.
# One can rename this script to LANG (or using symlink) and put it in PATH, if putting the script next to 
# the cmake executable is not doable.


usage() {
    printf \
"ninja: FATAL: $1\n\nusage:\n"\
"cmcldeps "\
"<language C, CXX or RC>  "\
"<source file path>  "\
"<output path for *.d file>  "\
"<output path for *.obj file>  "\
"<prefix of /showIncludes>  "\
"<path to cl.exe>  "\
"<path to tool (cl or rc)>  "\
"<rest of command ...>\n"\
    >&2
    exit 1
}

outputDepFile() {
    local dfile=$1   ; shift
    local objfile=$1 ; shift

    if [[ -z $dfile ]]; then
        return
    fi

    mapfile -t incs < <(printf '%s\n' "$@" | sort -u)
    
    local cwd="$(pwd)/"
    printf '%q \\\n' "$objfile:" "${incs[@]#$cwd}" >$dfile
}

process() {
    local srcfilename=$1    ; shift
    local dfile=$1          ; shift
    local objfile=$1        ; shift
    local prefix=$1         ; shift
    local cmd=$1            ; shift
    local dir=$1            ; shift
    local quiet=${1:-false} ; shift

    exec {fd}< <([[ -n $dir ]] && cd "$dir"; eval $cmd 2>&1)
    local pid=$!
    mapfile -t -u $fd
    wait $pid
    local exit_code=$?
    exec {fd}<&-

    # process the include directives and output everything else
    local includes=()
    local isFirstLine=true # cl prints always first the source filename
    for inc in "${MAPFILE[@]}"; do
        if [[ $inc =~ ^$prefix[[:blank:]]*(.*)$ ]]; then
            includes+=("${BASH_REMATCH[1]}")
        else
            if ! $isFirstLine || [[ $inc != ${srcfilename}* ]]; then
                if ! $quiet || [[ $exit_code -ne 0 ]]; then
                    printf '%s\n' "$inc"
                fi
            else
                isFirstLine=false
            fi
        fi
    done

    # don't update .d until/unless we succeed compilation
    if [[ $exit_code -eq 0 ]]; then
        outputDepFile "$dfile" "$objfile" "${includes[@]}"
    fi

    return $exit_code
}


lang=$(basename "$0")
if [[ $lang == cmcldeps ]]; then
    lang=$1;    shift
fi

srcfile=$1; shift
dfile=$1;   shift
objfile=$1; shift
prefix=$1;  shift
cl=$1;      shift
binpath=$1; shift
rest=("$@")

if [[ -z $binpath ]]; then
    usage "Couldn't parse arguments."
fi

# https://github.com/ninja-build/ninja/blob/v1.11.1/src/clparser.cc#L46
if [[ -z $prefix ]]; then
    prefix="Note: including file: "
fi

# needed to suppress filename output of msvc tools
srcfilename=$(basename "$srcfile")

if [[ $lang == C || $lang == CXX ]]; then

    process "$srcfilename" "$dfile" "$objfile" "$prefix" "'$binpath' /nologo /showIncludes ${rest[*]@Q}"

elif [[ $lang == RC ]]; then
    # "misuse" cl.exe to get headers from .rc files

    clrest=()
    for a in "${rest[@]}"; do
        case $a in
            [-/]fo | *$objfile) ;;
            *) clrest+=("$a") ;;
        esac
    done

    # call cl in object dir so the .i is generated there
    objdir=$(dirname "$objfile")

    # extract dependencies with cl.exe
    process "$srcfilename" "$dfile" "$objfile" "$prefix" "'$cl' /P /DRC_INVOKED /TC /nologo /showIncludes ${clrest[*]@Q}" "$objdir" true

    exit_code=$?
    if [[ $exit_code -ne 0 ]]; then
        exit $exit_code
    fi

    # compile rc file with rc.exe
    process "$srcfilename" "" "$objfile" "$prefix" "'$binpath' ${rest[*]@Q}" "" true

else
    usage "Invalid language specified."
fi

May 12 '23 16:05 huangqinjin

Nice hacks with reimplementing this in bash!

I took a look at the cmcldeps tool itself, and it doesn't seem all too reliant on Windows specific APIs, so I made an attempt at tweaking it so that it can be built and used on all platforms. See https://gitlab.kitware.com/mstorsjo/cmake/-/commits/cmcldeps. Note, these patches are entirely untested (I've only tested that it does compile), but if you're interested in progressing on this issue, you can have a look at this - hopefully it's not too far away from getting it to work.

May 16 '23 11:05 mstorsjo

Nice! Regarding rc: /fo x.dir\x.rc.res -> cl: /out:x.dir\x.rc.res.dep.obj, I have reported https://gitlab.kitware.com/cmake/cmake/-/issues/24906, /fo x.dir\x.rc.res should be removed when invoking cl.exe. Hopefully this makes your work slightly simpler.

P.S. It would be better to merge cmcldeps into cmake finally (if cmcldeps works on all OSes), just like vs_link_exe and vs_link_dll.

May 16 '23 11:05 huangqinjin