xmake-io
xmake includes are always being built even if --shallow is used
Xmake Version
2.9.9
Operating System Version and Architecture
Windows 11 Pro Version 10.0.26100 Build 26100
Describe Bug
I have a toy engine that uses xmake for its module management. Some modules have dependencies to core modules that are already loaded by the engine, those are being defined with 'add_deps' in the xmake.lua files.
The problem I am facing is that building a module is trying to build the dependency target even if it's already built:
Compiling module: VulkanBackend
System Command: xmake build --shallow -F "D:/Projects/Ravine Projects/RavineEngine/modules//VulkanBackend/xmake.lua" VulkanBackend
[ 64%]: compiling.debug datagen\src\vulkanBackend_DataGen.cpp
[ 64%]: compiling.debug src\private\imguiImpl.cpp
[ 64%]: compiling.debug src\private\imguiPipeline.cpp
[ 64%]: compiling.debug src\private\imgui_impl_glfw.cpp
[ 64%]: compiling.debug src\private\meshAllocator.cpp
[ 64%]: compiling.debug src\vulkanBackend.cpp
[ 78%]: linking.debug CoreRendering.dll
create ok!
compile_commands.json updated!
error: LINK : fatal error LNK1168: cannot open ..\..\bin\Debug\CoreRendering.dll for writing
Compilation failed!
Expected Behavior
I expected it not to try to link the DLL again considering I am using the --shallow flag when building. Moreover, I expected it not to build the DLL again even without the flag because nothing changed in the compile commands - and this is working fine on MacOS (although I only tested the previous release of xmake).
Project Configuration
/modules/CoreRendering/xmake.lua
add_rules("plugin.vsxmake.autoupdate")
add_rules("plugin.compile_commands.autoupdate")
add_rules("mode.debug", "mode.release")
add_requires("glm", "tinygltf", "fmt")
-- Engine dependency
includes("../../.")
target("CoreRendering")
set_kind("shared")
set_languages("c++20")
add_deps("Ravine")
-- TODO: Handle static linkage option
add_defines("RENDERING_EXPORTS")
add_defines("USE_STL_LIB")
-- Public Packages
add_packages("glm", {public = true, header_only = true})
add_packages("tinygltf", {public = true})
-- Private Packages
add_packages("fmt")
add_includedirs("src/", "src/public/", {public = true})
add_headerfiles("src/public/*.h", "src/*.h", {public = true})
add_includedirs("src", "src/private/")
add_headerfiles("src/private/*.h")
add_files("src/**.cpp")
if is_mode("debug") then
if is_plat("windows") then
add_cxxflags("/Od")
else
add_cxxflags("-O0")
end
add_defines("RV_DEBUG_BUILD")
set_targetdir("../../bin/Debug")
elseif is_mode("release") then
add_defines("RV_RELEASE_BUILD")
set_targetdir("../../bin/Release")
end
target_end()
/modules/VulkanBackend/xmake.lua
add_rules("plugin.vsxmake.autoupdate")
add_rules("plugin.compile_commands.autoupdate")
add_rules("mode.debug", "mode.release")
-- Third-party dependencies
add_requires("glfw", {configs = {shared = true}})
add_requires("glslang", {configs = {shared = true}})
add_requires("imgui v1.91.0-docking", {configs = {header_only = true}})
add_requires("vulkan-memory-allocator", {configs = {header_only = true}})
add_requires("volk", "fmt", "glm", "glad", "vulkan-headers")
-- Include engine project
includes('../../.')
-- Include datagen dependencies
includes('./datagen')
-- Module dependencies
includes("../CoreSimulation")
includes("../CoreRendering")
target("VulkanBackend")
set_kind("shared")
add_deps("CoreSimulation")
add_deps("CoreRendering")
-- Engine dependencies
add_deps("Ravine")
-- DataGen dependencies
add_deps("VulkanBackend_DataGen")
add_packages("glm", "glfw", "glad", "fmt", "imgui", "glslang", "vulkan-headers", "volk", "vulkan-memory-allocator")
add_includedirs("src", "src/public/", "src/private/")
add_headerfiles("src/**.h")
add_files("src/**.cpp")
set_languages("c++20")
add_defines("USE_STL_LIB")
-- add_defines("USE_EASTL_LIB")
if is_mode("debug") then
if is_plat("windows") then
add_cxxflags("/Od")
else
add_cxxflags("-O0")
end
add_defines("RV_DEBUG_BUILD")
set_targetdir("../../bin/Debug/")
elseif is_mode("release") then
add_defines("RV_RELEASE_BUILD")
set_targetdir("../../bin/Release/")
end
-- add_defines("RV_EMIT_SHADERS_DEBUG_INFO")
-- Add rpath to the target (required for shared libraries in Linux and MacOS)
if is_plat("linux") then
add_rpathdirs("$ORIGIN")
elseif is_plat("macosx") then
add_rpathdirs("@executable_path")
end
-- Post-build step to copy data folder to target directory (for shaders)
after_build(function (target)
local pkgs = {target:pkg("glfw"), target:pkg("glslang")};
for _, pkg in ipairs(pkgs) do
print(pkg:installdir())
print("Copying shared libraries to target directory...")
os.cp(pkg:installdir() .. "/bin/*.dll", target:targetdir()) -- Windows
os.cp(pkg:installdir() .. "/lib/*.so*", target:targetdir()) -- Linux
os.cp(pkg:installdir() .. "/lib/*.dylib", target:targetdir()) -- MacOS
end
print("Copying data folder to target directory...")
local targetDir = target:targetdir()
os.cp("data", targetDir .. "/../", {force = true})
print("Done!")
end)
target_end()
Additional Information and Error Logs
The console command runs from this C++ logic:
// Compile module
{
String compileCmd = "xmake build --shallow -F " + moduleXmakePath + " " + moduleName;
std::cout << "\nCompiling module:\t" << moduleName << "\n";
std::cout << "System Command:\t\t" << compileCmd << "\n";
std::flush(std::cout);
if (system(compileCmd.c_str()) == 0)
{
std::cout << "Compilation succeeded!\n\n";
std::flush(std::cout);
}
else
{
std::cout << "Compilation failed!\n\n";
std::flush(std::cout);
continue;
}
}
Here is the output:
Compiling module: VulkanBackend
System Command: xmake build --shallow -F "D:/Projects/Ravine Projects/RavineEngine/modules//VulkanBackend/xmake.lua" VulkanBackend
[ 64%]: compiling.debug datagen\src\vulkanBackend_DataGen.cpp
[ 64%]: compiling.debug src\private\imguiImpl.cpp
[ 64%]: compiling.debug src\private\imguiPipeline.cpp
[ 64%]: compiling.debug src\private\imgui_impl_glfw.cpp
[ 64%]: compiling.debug src\private\meshAllocator.cpp
[ 64%]: compiling.debug src\vulkanBackend.cpp
[ 78%]: linking.debug CoreRendering.dll
create ok!
compile_commands.json updated!
error: LINK : fatal error LNK1168: cannot open ..\..\bin\Debug\CoreRendering.dll for writing
Compilation failed!
It works for me. Please provide a minimal reproducible complete project.
PS C:\Users\wangrunqing\Downloads> xmake create -t shared testdll
create testdll ...
[+]: src\foo.cpp
[+]: src\foo.h
[+]: src\main.cpp
[+]: xmake.lua
[+]: .gitignore
create ok!
PS C:\Users\wangrunqing\Downloads> cd testdll
PS C:\Users\wangrunqing\Downloads\testdll> xmake
checking for platform ... windows
checking for architecture ... x64
checking for Microsoft Visual Studio (x64) version ... 2022
checking for Microsoft C/C++ Compiler (x64) version ... 19.44.34823.2
[ 23%]: compiling.release src\main.cpp
[ 23%]: compiling.release src\foo.cpp
[ 35%]: linking.release foo.dll
[ 71%]: linking.release testdll.exe
[100%]: build ok, spent 0.984s
PS C:\Users\wangrunqing\Downloads\testdll> xmake -r testdll
[ 17%]: compiling.release src\main.cpp
[ 23%]: compiling.release src\foo.cpp
[ 35%]: linking.release foo.dll
[ 71%]: linking.release testdll.exe
[100%]: build ok, spent 0.735s
PS C:\Users\wangrunqing\Downloads\testdll> xmake -r --shallow testdll
[ 17%]: compiling.release src\main.cpp
[ 71%]: linking.release testdll.exe
[100%]: build ok, spent 0.969s
In my observe
1.--shallow is used together with -r
2.--shallow -r is used to force rebuild specifical target and it's depency targets are build by depend
3.Use policy set_policy("diagnosis.check_build_deps", true) to see what depend changed
4.So you might not need this at all
Sorry, I can't provide a minimal reproducible example atm - this is a hobby project and I don't get to work on it that often. I will try to provide one later this week.
I ran the build with the set_policy("diagnosis.check_build_deps", true) as @wagcheg mentioned and got some interesting results:
Compiling module: VulkanBackend
System Command: xmake build -F "D:/Projects/Ravine Projects/RavineEngine/modules//VulkanBackend/xmake.lua" VulkanBackend
[ 67%]: compiling.debug src\private\imguiImpl.cpp
[check_build_deps]: file datagen\src\vulkanBackend_DataGen.cpp is changed, mtime: 1744415911, lastmtime: 0
[ 78%]: compiling.debug datagen\src\vulkanBackend_DataGen.cpp
[ 78%]: compiling.debug src\vulkanBackend.cpp
[ 78%]: compiling.debug src\private\meshAllocator.cpp
[ 78%]: compiling.debug src\private\imguiPipeline.cpp
[check_build_deps]: file ..\..\bin\Debug\build\.objs\CoreRendering\windows\x64\debug\src\coreRendering.cpp.obj != ..\..\bin\Debug\build\.objs\CoreRendering\windows\x64\debug\__\CoreRendering\src\coreRendering.cpp.obj at index 1
[ 82%]: linking.debug CoreRendering.dll
create ok!
compile_commands.json updated!
error: LINK : fatal error LNK1168: cannot open ..\..\bin\Debug\CoreRendering.dll for writing
When comparing both files here is what I see:
Since one is being built as a dependency (and the inclusion uses a relative path) the .obj file gets generated using relative paths too... Aside from that, I don't know what the depfiles_format = "cl_json" portion means. I tried --shallow -r and it didn't work (same issue).
@waruqi in your example, is the executable running? My engine builds and dynamically loads the DLLs soon after. Then, later, when initializing the Rendering code it builds the VulkanBackend module (which tries to override the DLL already loaded by the engine). I am running the CLI command from my executable which has the DLL loaded.
I will make a new repo with a minimal repro case. Thanks to both for your answers so far!
I realized I could work around this problem by making a copy of the DLLs and dynamically loading those instead of the ones the compiler will write to...
Do you think this problematic use case is valid for xmake to support? I don't see why the DLL would have to be linked again in this case, so adjustments to the logic could be made as an improvement.