dub git-dependency testcase is broken on Windows

System information

dub version: dub (master branch / commit: a8aa2341be8434b16630aba03958f70565ff3ad6)
OS Platform and distribution: Windows 11
compiler version dmd-2.100.0 / ldc-1.30.0

Bug Description

test/git-dependency is not work on Windows after https://github.com/dlang/dub/pull/2386

This problem occurs only on Windows. The contents of test/git-dependency test case show that it can be built on Windows. It is also a regression, as it builds successfully on past versions. The point at which the problem occurred was tracked down by git bisect.

How to reproduce?

git checkout a8aa2341be8434b16630aba03958f70565ff3ad6
dub run -- build --root=test/git-dependency

Expected Behavior

Successful builds, even on Windows

Logs

Log

R:\dub> dub run -- build --root .\test\git-dependency -v
Performing "debug" build using P:\app\dmd\bin64\dmd.exe for x86_64.
dub ~master: building configuration "application"...
Serializing composite type Flags!(BuildRequirement) which has no serializable fields
Serializing composite type Flags!(BuildOption) which has no serializable fields
Linking...
Running bin/dub.exe build --root .\test\git-dependency -v
Using dub registry url 'https://code.dlang.org/'
Refreshing local packages (refresh existing: true)...
......
......
......
Looking for local package map at R:\dub\test\git-dependency\.dub\packages\local-packages.json
  Version selection for dependency gitcompatibledubpackage (gitcompatibledubpackage) of git-dependency is missing.
  Found dependency gitcompatibledubpackage git+https://github.com/dlang-community/gitcompatibledubpackage.git
Generating using build
Configuring dependent git-dependency, deps:"gitcompatibledubpackage"
  Configuring dependent gitcompatibledubpackage, deps:
    Starting Performing "debug" build using dmd for x86_64.
Target '......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage\.dub\build\lib-debug-windows-x86_64-dmd_v2.100.1-987A8C41F524B24BEFD436F70852ACB9ED35FAEA102BE102629C2BEA03A44814\gitcompatibledubpackage.lib' doesn't exist, need rebuild.
             Build directory ......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage\.dub\build\lib-debug-windows-x86_64-dmd_v2.100.1-987A8C41F524B24BEFD436F70852ACB9ED35FAEA102BE102629C2BEA03A44814\ is not writable. Falling back to direct build in the system's temp folder.
    Building gitcompatibledubpackage 1.0.4+commit.2.gccb31bf [lib]
dmd -m64 -lib -of......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage\gitcompatibledubpackage.lib -debug -g -w -version=Have_gitcompatibledubpackage -I......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67 ......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage\subdir\file.d -vcolumns
Target 'R:\dub\test\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-5C3FC3BA134840FB42131EF1DCE7C916264FCEB8161FDA0557EA48231B93C420\git-dependency.exe' doesn't exist, need rebuild.
    Building git-dependency ~master: building configuration [application]
dmd -m64 -c -oftest\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-5C3FC3BA134840FB42131EF1DCE7C916264FCEB8161FDA0557EA48231B93C420\git-dependency.obj -debug -g -w -version=Have_git_dependency -version=Have_gitcompatibledubpackage -Itest\git-dependency\src -I......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67 test\git-dependency\src\app.d -vcolumns
     Linking git-dependency
dmd -oftest\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-5C3FC3BA134840FB42131EF1DCE7C916264FCEB8161FDA0557EA48231B93C420\git-dependency.exe test\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-5C3FC3BA134840FB42131EF1DCE7C916264FCEB8161FDA0557EA48231B93C420\git-dependency.obj -m64 -g
git-dependency.obj : error LNK2019: 未解決の外部シンボル _D23gitcompatibledubpackage6subdir4file19hasTheWorldExplodedFZb が関数 _Dmain で参照されました。
test\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-5C3FC3BA134840FB42131EF1DCE7C916264FCEB8161FDA0557EA48231B93C420\git-dependency.exe : fatal error LNK1120: 1 件の未解決の外部参照
Error: linker exited with status 1120
FAIL test\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-5C3FC3BA134840FB42131EF1DCE7C916264FCEB8161FDA0557EA48231B93C420\ git-dependency executable
Error dmd failed with exit code 1.
Program exited with code 2

Log before #2386

R:\dub> git checkout 71ae566bc2630b73e386def15eb2f5bbb494ade1 -q
R:\dub> dub run -- build --root .\test\git-dependency -v
Performing "debug" build using P:\app\dmd\bin64\dmd.exe for x86_64.
dub 1.29.1+commit.215.g71ae566b: building configuration "application"...
Serializing composite type Flags!(BuildRequirement) which has no serializable fields
Serializing composite type Flags!(BuildOption) which has no serializable fields
Linking...
Running bin/dub.exe build --root .\test\git-dependency -v
Using dub registry url 'https://code.dlang.org/'
Refreshing local packages (refresh existing: true)...
......
......
......
Looking for local package map at R:\dub\test\git-dependency\.dub\packages\local-packages.json
  Version selection for dependency gitcompatibledubpackage (gitcompatibledubpackage) of git-dependency is missing.
  Found dependency gitcompatibledubpackage git+https://github.com/dlang-community/gitcompatibledubpackage.git
Generating using build
Configuring dependent git-dependency, deps:"gitcompatibledubpackage"
  Configuring dependent gitcompatibledubpackage, deps:
    Starting Performing "debug" build using dmd for x86_64.
  Up-to-date gitcompatibledubpackage 1.0.4+commit.2.gccb31bf: target for configuration [lib] is up to date.
Using existing build in ......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage\.dub\build\lib-debug-windows-x86_64-dmd_v2.100.1-3B8E4AEDEEC3AEE060DBE78BC72E6118\.
Copying target from ......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage\.dub\build\lib-debug-windows-x86_64-dmd_v2.100.1-3B8E4AEDEEC3AEE060DBE78BC72E6118\gitcompatibledubpackage.lib to ......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage
Target 'R:\dub\test\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-D791E20AFF2E02A675F577B266B98B87\git-dependency.exe' doesn't exist, need rebuild.
    Building git-dependency ~master: building configuration [application]
dmd -m64 -c -oftest\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-D791E20AFF2E02A675F577B266B98B87\git-dependency.obj -debug -g -w -version=Have_git_dependency -version=Have_gitcompatibledubpackage -Itest\git-dependency\src -I......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67 test\git-dependency\src\app.d -vcolumns
     Linking git-dependency
dmd -oftest\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-D791E20AFF2E02A675F577B266B98B87\git-dependency.exe test\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-D791E20AFF2E02A675F577B266B98B87\git-dependency.obj ......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage\.dub\build\lib-debug-windows-x86_64-dmd_v2.100.1-3B8E4AEDEEC3AEE060DBE78BC72E6118\gitcompatibledubpackage.lib -m64 -g
Copying target from R:\dub\test\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-D791E20AFF2E02A675F577B266B98B87\git-dependency.exe to R:\dub\test\git-dependency
Copying target from R:\dub\test\git-dependency\.dub\build\application-debug-windows-x86_64-dmd_v2.100.1-D791E20AFF2E02A675F577B266B98B87\git-dependency.pdb to R:\dub\test\git-dependency
Copying target from ......\AppData\Local\dub\packages\gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67\gitcompatibledubpackage\.dub\build\lib-debug-windows-x86_64-dmd_v2.100.1-3B8E4AEDEEC3AEE060DBE78BC72E6118\gitcompatibledubpackage.lib to R:\dub\test\git-dependency
    Finished To force a rebuild of up-to-date targets, run again with --force

Log on linux(ubuntu)

~/dub# dub run -- clean --root ./test/git-dependency -v
Performing "debug" build using /dlang/dmd-2.100.0/linux/bin64/dmd for x86_64.
dub 1.29.1+commit.260.ga8aa2341: target for configuration "application" is up to date.
To force a rebuild of up-to-date targets, run again with --force.
Running bin/dub clean --root ./test/git-dependency -v
Using dub registry url 'https://code.dlang.org/'
Refreshing local packages (refresh existing: true)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at /dlang/dub/packages/local-packages.json
Looking for local package map at /root/dub/test/git-dependency/.dub/packages/local-packages.json
Determined package version using GIT: gitcompatibledubpackage 1.0.4+commit.2.gccb31bf
Note: Failed to determine version of package git-dependency at .. Assuming ~master.
    Cleaning package at /root/dub/test/git-dependency
Note: Failed to determine version of package git-dependency at .. Assuming ~master.
root@b6290cb61961:~/dub# dub run -- build --root ./test/git-dependency -v
Performing "debug" build using /dlang/dmd-2.100.0/linux/bin64/dmd for x86_64.
dub 1.29.1+commit.260.ga8aa2341: target for configuration "application" is up to date.
To force a rebuild of up-to-date targets, run again with --force.
Running bin/dub build --root ./test/git-dependency -v
Using dub registry url 'https://code.dlang.org/'
Refreshing local packages (refresh existing: true)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at /dlang/dub/packages/local-packages.json
Looking for local package map at /root/dub/test/git-dependency/.dub/packages/local-packages.json
Determined package version using GIT: gitcompatibledubpackage 1.0.4+commit.2.gccb31bf
Note: Failed to determine version of package git-dependency at .. Assuming ~master.
Refreshing local packages (refresh existing: false)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at /dlang/dub/packages/local-packages.json
Looking for local package map at /root/dub/test/git-dependency/.dub/packages/local-packages.json
  Version selection for dependency gitcompatibledubpackage (gitcompatibledubpackage) of git-dependency is missing.
  Found dependency gitcompatibledubpackage git+https://github.com/dlang-community/gitcompatibledubpackage.git
Generating using build
Configuring dependent git-dependency, deps:"gitcompatibledubpackage"
  Configuring dependent gitcompatibledubpackage, deps:
    Starting Performing "debug" build using dmd for x86_64.
  Up-to-date gitcompatibledubpackage 1.0.4+commit.2.gccb31bf: target for configuration [lib] is up to date.
Using existing build in /dlang/dub/packages/gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67/gitcompatibledubpackage/.dub/build/lib-debug-linux.posix-x86_64-dmd_v2.100.0-2BD2AD6CE46F3CB081C4D45BADB78A189DE72B135C568890B14BAA3CBE87FE3D/.
Copying target from /dlang/dub/packages/gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67/gitcompatibledubpackage/.dub/build/lib-debug-linux.posix-x86_64-dmd_v2.100.0-2BD2AD6CE46F3CB081C4D45BADB78A189DE72B135C568890B14BAA3CBE87FE3D/libgitcompatibledubpackage.a to /dlang/dub/packages/gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67/gitcompatibledubpackage
Target '/root/dub/test/git-dependency/.dub/build/application-debug-linux.posix-x86_64-dmd_v2.100.0-46A626078465576416FA921BC9D36AA8C0C7D6654F16266056103E0FFE030FA4/git-dependency' doesn't exist, need rebuild.
    Building git-dependency ~master: building configuration [application]
dmd -c -oftest/git-dependency/.dub/build/application-debug-linux.posix-x86_64-dmd_v2.100.0-46A626078465576416FA921BC9D36AA8C0C7D6654F16266056103E0FFE030FA4/git-dependency.o -debug -g -w -version=Have_git_dependency -version=Have_gitcompatibledubpackage -Itest/git-dependency/src/ -I/dlang/dub/packages/gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67 test/git-dependency/src/app.d -vcolumns
     Linking git-dependency
dmd -oftest/git-dependency/.dub/build/application-debug-linux.posix-x86_64-dmd_v2.100.0-46A626078465576416FA921BC9D36AA8C0C7D6654F16266056103E0FFE030FA4/git-dependency test/git-dependency/.dub/build/application-debug-linux.posix-x86_64-dmd_v2.100.0-46A626078465576416FA921BC9D36AA8C0C7D6654F16266056103E0FFE030FA4/git-dependency.o /dlang/dub/packages/gitcompatibledubpackage-ccb31bf6a655437176ec02e04c2305a8c7c90d67/gitcompatibledubpackage/.dub/build/lib-debug-linux.posix-x86_64-dmd_v2.100.0-2BD2AD6CE46F3CB081C4D45BADB78A189DE72B135C568890B14BAA3CBE87FE3D/libgitcompatibledubpackage.a -L--no-as-needed -g
Copying target from /root/dub/test/git-dependency/.dub/build/application-debug-linux.posix-x86_64-dmd_v2.100.0-46A626078465576416FA921BC9D36AA8C0C7D6654F16266056103E0FFE030FA4/git-dependency to /root/dub/test/git-dependency
    Finished To force a rebuild of up-to-date targets, run again with --force

Logs have been partially processed.

Aug 28 '22 05:08 shoo

Perhaps this happened because the length of the path exceeded MAX_PATH(260)?

Sep 02 '22 16:09 shoo

I am suspicious of the following line: https://github.com/dlang/dub/blob/32985ea8aae88a8d50d68f9d25b7c8163c69abce/source/dub/generators/build.d#L242

On Windows, if MAX_PATH(260) is exceeded, it is determined that writing is not possible and falling back to direct build in the system's temp folder. The change in #2386 resulted in the creation of an intermediate folder with SHA256, which exceeded the MAX_PATH constraint, and the above process is now performed. At this time, performDirectBuild is called and then the artifact must be stored in target_binary_path. However, target_path is specified in the function argument, and the artifact is not correctly propagated to the caller.

There are three solutions to this problem:

Fix the fallback process to work properly as described above
Make sure it builds correctly even if MAX_PATH is exceeded (this measure is actually difficult because the Visual Studio linker is unsupported)
Make sure that MAX_PATH is not exceeded. In other words, the naming convention for intermediate folders must be changed.

In the short term, 1 may need to be fixed, but in the long term, I would prefer to fix 2 or 3. Is my guess likely to be correct? And are there any other measures you can think of?

Sep 04 '22 13:09 shoo

Thanks for the great writeup!

Is my guess likely to be correct? And are there any other measures you can think of?

Likely, although I'm not sure why you mentioned the VS Linker is unsupported ? Also, 2 is definitely the way to go.

Sep 05 '22 03:09 Geod24

Likely, although I'm not sure why you mentioned the VS Linker is unsupported ?

The following commands cannot be linked cause of the path length:

R:\>"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.22.27905\bin\HostX64\x64\link.exe" /NOLOGO "R:\abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij\abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij\abcdefghijabcdefghijabcdefghijabcdefghijababcdefghij\main.obj" /DEFAULTLIB:phobos64.lib /LIBPATH:"P:\app\dmd\lib64" /LIBPATH:"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.22.27905\lib\x64" legacy_stdio_definitions.lib /LIBPATH:"C:\Program Files (x86)\Windows Kits\10\Lib\10.0.18362.0\ucrt\x64" /LIBPATH:"C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64"
LINK : fatal error LNK1181: 入力ファイル '.obj' を開けません。

R:\abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij\abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij\abcdefghijabcdefghijabcdefghijabcdefghijababcdefghij>"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.22.27905\bin\HostX64\x64\link.exe" /NOLOGO "main.obj" /DEFAULTLIB:phobos64.lib /LIBPATH:"P:\app\dmd\lib64" /LIBPATH:"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.22.27905\lib\x64" legacy_stdio_definitions.lib /LIBPATH:"C:\Program Files (x86)\Windows Kits\10\Lib\10.0.18362.0\ucrt\x64" /LIBPATH:"C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64"
LINK : fatal error LNK1181: 入力ファイル 'main.obj' を開けません。

I haven't tried newer versions or lld-link, but at least the 2019 link.exe doesn't appear to work.

Sep 05 '22 13:09 shoo

fixed by #2589

Feb 25 '23 22:02 WebFreak001

dub dub copied to clipboard

git-dependency testcase is broken on Windows

System information

Bug Description

How to reproduce?

Expected Behavior

Logs

dub
dub copied to clipboard