Compile Line Length Limit hit under Windows

[adam-porich-sm]

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

The following build file does not compile under windows

cc_library(
    name = "long",
    includes = ["a" * 32768]
)
cc_binary(
    name = "long_bin",
    srcs = ["main.cpp"],
    deps = [
        ":long"
    ]
)

The reported error is Action failed to execute: java.io.IOException: ERROR: src/main/native/windows/processes-jni.cc(161): CreateProcessWithExplicitHandles("C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\amd64\cl.exe" /c main.cpp /Fobazel-out/x64_windows-fastbuild/bin/_objs/long_bin/main.o /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0(...)): command is longer than CreateProcessW's limit (32768 characters)

From what I could tell when trying to make a simple repo it appears this case may be handled correctly for link steps but not for compile steps.

What operating system are you running Bazel on?

Windows 10

What's the output of bazel info release?

release 0.13.0

[RNabel]

This is a known issue. Did you check whether https://github.com/bazelbuild/bazel/issues/4149 helps in your case?

[adam-porich-sm]

Good point, but unfortunately doesn't help in my case.

Under Windows there are 3 main relevant limitations

1. Filename paths can't be longer than MAX_PATH (260) characters
2. Command-line lengths can't be longer than 32,768 characters
3. Response-file lengths can't be longer than 131,072 characters

This example breaches both 1 and 2. But the problem in my actual code is breaching 2. We have many dependent libraries with manually set include paths (On the migration path to remove them, but we aren't there yet). This results in extremely long compiler command lines, which without de-duplication (which bazel handles nicely) will overflow the response-file limits breaching the third limitation.

There are two solutions that I see. One is that the manually adding include paths appears too add two include paths one for generated files which often don't exist. And the other is to support param files the same way they are supported for link command lines.

I think ideally in the long-run both solutions would be implemented. I had a go at the param files but due to bazel supporting things like remote execution I needed a better understanding of bazel internals than I currently have.

[laszlocsomor]

There are two solutions that I see. One is that the manually adding include paths appears too add two include paths one for generated files which often don't exist. And the other is to support param files the same way they are supported for link command lines.

I'd like to reply but I'm not sure I parse this correctly, could you please explain?

Let me /cc @tomlu who's been drastically improving the parameter file handling lately. Thanks to his work it'll be easier to create parameter files, which may interest you @adamporich.

[gimo]

The only real solution is supporting parameter file handling for C++ compiler arguments. Which is currently supported for C++ linkers.

Also, a significant mitigation for the issue is that for every "includes" entry in a "cc_library" invocation I get two directories added to the compiler command line. One for the include directory in the source folder and one mirror into the "genfiles" tree. In most cases for me a library has no "genfiles" headers and so the second entry is irrelevant.

[laszlocsomor]

I see, thanks for the clarification!

The only real solution is supporting parameter file handling for C++ compiler arguments. Which is currently supported for C++ linkers.

Do you mean VC++ linker or Clang/GCC? FYI VC++ is the default compiler on Windows. Do you know whether its compiler supports parameter files? If so, then @tomlu 's work will enable updating the C++ rules so they can create parameter files on demand. (Adding @meteorcloudy, the Windows C++ compilation expert and @mhlopko the overall expert on C++ rules.)

Also, a significant mitigation for the issue is that for every "includes" entry in a "cc_library" invocation I get two directories added to the compiler command line. One for the include directory in the source folder and one mirror into the "genfiles" tree. In most cases for me a library has no "genfiles" headers and so the second entry is irrelevant.

Right, I understand that that's an unnecessary limitation for you. I'm afraid Bazel is not (yet?) smart enough to put the genfiles directory on the inclusion path only when it needs to, but maybe @mhlopko knows about an effort to change that. In any case, is it an option for you to split the libraries so they need fewer includes and therefore shorter includes lists?

[meteorcloudy]

Yes, I think a parameter file for C++ compilation action should be the solution. @mhlopko We already have a feature called linker_param_file, do you think we can also have a compiler_param_file? The MSVC compiler does support parameter file. See https://msdn.microsoft.com/en-us/library/610ecb4h.aspx

[laszlocsomor]

@meteorcloudy : Thanks! Shall I categorize this bug as a feature request then? I reckon compiler_param_file doesn't exist yet.

[meteorcloudy]

@laszlocsomor Please do that~

[laszlocsomor]

And if I'm not mistaken, this isn't a Windows bug. Removing the Windows tag and assigning to @mhlopko for now. Please reassign if there's a better assignee.

[hlopko]

Late is better than never.

We changed includes handling and we will now add 3 include directories per single include (srcs, genfiles, binfiles). I don't plan to optimize this in the short term.

We have param files for C++ link actions, not yet for compile actions. Unfortunately work that @tomlu did does not apply to C++ actions since they are not standard SpawnActions but use FeatureConfiguration and CcToolchainVariables to construct command lines.

What's not clear to me is which flags can go to param files. For link actions there is a heuristic that decides which flags go where. This is even worse than it looks like, we have crosstools that rely on the change of order param files introduce. Ideally all flags would go to param file, but the internal migration won't be trivial.

We are not burdened by the migration for param files for compile actions, but I'd prefer if we first solved the linking case and use the same solution for compilation and archiving. How quickly do we need to solve this?

[adam-porich-sm]

We should be able to workaround the issues in Windows for now and the limits are more relaxed under Linux.

I must confess at being a bit confused by that heuristic though. Does it not actually work to pass the arguments straight through. It's been a while since I have tried but I thought MSVC supported all arguments being passed through param files.

[tomlu]

On Tue, May 15, 2018 at 2:07 AM, Marcel Hlopko [email protected] wrote:

Late is better than never.

We changed includes handling and we will now add 3 include directories per single include (srcs, genfiles, binfiles). I don't plan to optimize this in the short term.

We have param files for C++ link actions, not yet for compile actions. Unfortunately work that @tomlu https://github.com/tomlu did does not apply to C++ actions since they are not standard SpawnActions but use FeatureConfiguration and CcToolchainVariables to construct command lines.

It does apply, if we rewrite the C++ actions to use CommandLines the same way SpawnAction does. It's not hard, the action just has to store CommandLines then expand them when it builds the Spawn, instead of manually constructing ParamFileWriteActions.

Unfortunately I don't touch C++ actions generally because they are too complicated.

What's not clear to me is which flags can go to param files. For link actions there is a heuristic https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/rules/cpp/LinkCommandLine.java;l=287?q=linkcommandline that decides which flags go where.

This is the part of "I can't touch this because it's too complicated".

It's unclear to me why we can't build two command lines instead of building one, then iterating over it and separating stuff out.

This is even worse than it looks like, we have crosstools that rely on the change of order param files introduce. Ideally all flags would go to param file, but the internal migration won't be trivial.

We are not burdened by the migration for param files for compile actions, but I'd prefer if we first solved the linking case and use the same solution for compilation and archiving. How quickly do we need to solve this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bazelbuild/bazel/issues/5163#issuecomment-389053276, or mute the thread https://github.com/notifications/unsubscribe-auth/ABbnSvHu3LtoLhzLxGVS4rctePTcD6b7ks5tynCogaJpZM4TzjQC .

[sesmith177]

we are working on porting a large bazel c++ project (https://github.com/envoyproxy/envoy/issues/129) to Windows and are running into this issue (cl.exe command line larger than 32,768 characters) as well

[hlopko]

I wrote a doc proposing unifying C++ actions with Args (what all other actions use): https://docs.google.com/document/d/11DDjHQfCY3I1kO7w0PusjCoaqmZUQQajd42HDS0F3dI/edit#heading=h.4nax2orizccp

[sesmith177]

Is there a time frame in the road map for this to be fixed? We are working around the issue now by forking Bazel and changing the "_virtual_includes" part of the path to something shorter.

[hlopko]

We discussed this offline with @laszlocsomor, and he'll ping this issue once he's done with his investigations with his plan.

[natalieparellano]

Hey @mhlopko @laszlocsomor any progress on this? The workaround we are using to compile Envoy may be on its last legs, as we are getting close to the character limit again even after making "_virtual_includes" as short as possible.

[laszlocsomor]

@natalieparellano : sorry for the silence, both @mhlopko and I have been away. Unfortunately we made no progress on this issue.

[sesmith177]

This issue is still causing us problems when building Envoy on Windows. In addition to the workaround of renaming the "_virtual_includes" directory to "v", we also find it necessary to strip off non-existent include directories from the command line. (our non-portable hack: https://github.com/greenhouse-org/bazel/commit/876b1995e70f1ad4e35d9f6cad5f599499a2349e)

So we're not 100% blocked right now, but we don't know for how long our workarounds will continue to do the trick

[laszlocsomor]

@sesmith177 : thanks for the update! @lberki, @mhlopko : what's your timeline to fix this issue?

[hlopko]

We plan to work on this in the first half of 2019. Definitely before Bazel 1.0 is released.

[oquenchil]

Try passing --features=compiler_param_file. Please let me know if this works.

[laszlocsomor]

Thanks! Could you please try this on Windows? There's a repro in the first comment.

[wrowe]

@oquenchil @laszlocsomor We've just checked out this workaround under our Apr 1st fork of envoy, using contemporaneous bazel 0.24.1 and the --features=compiler_param_file flag was not recognized/failed as a startup flag in BAZEL_BASE_OPTIONS, and was ignored in the list of BAZEL_BUILD_OPTIONS. Can you confirm if this feature was to work back in bazel 0.24.1 (Apr 2nd release) ERROR: C:/workspace/envoy/source/server/BUILD:263:1: C++ compilation of rule '//source/server:server_lib' failed (Exit -1). Note: Remote connection/protocol failed with: execution failed: cl.exe failed: error executing command cd C:/eb/execroot/envoy SET INCLUDE=C:\var\vcap\data\VSBuildTools\2017\VC\Tools\MSVC\14.16.27023\include;C:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\ucrt;C:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\shared;C:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\winrt;C:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\cppwinrt SET PATH=C:\var\vcap\data\VSBuildTools\2017\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64;C:\var\vcap\data\VSBuildTools\2017\MSBuild\15.0\bin\Roslyn;C:\Program Files (x86)\Windows Kits\10\bin\10.0.17134.0\x64;C:\Program Files (x86)\Windows Kits\10\bin\x64;C:\var\vcap\data\VSBuildTools\2017\MSBuild\15.0\bin;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\var\vcap\data\VSBuildTools\2017\Common7\IDE;C:\var\vcap\data\VSBuildTools\2017\Common7\Tools;;C:\Windows\system32 SET PWD=/proc/self/cwd SET TEMP=C:\Windows\TEMP SET TMP=C:\Windows\TEMP C:/var/vcap/data/VSBuildTools/2017/VC/Tools/MSVC/14.16.27023/bin/HostX64/x64/cl.exe /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0601 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_SECURE_NO_WARNINGS /bigobj /Zm500 /EHsc /wd4351 /wd4291 /wd4250 /wd4996 /I. /Ibazel-out/x64_windows-dbg/genfiles /Ibazel-out/x64_windows-dbg/bin /Iexternal/com_google_absl /Ibazel-out/x64_windows-dbg/genfiles/external/com_google_absl /Ibazel-out/x64_windows-dbg/bi .......... /Iexternal/com_github_grpc_grpc/third_party/address_sorting/include /Ibazel-out/x64_windows-dbg/genfiles/external/com_github_grpc_grpc/third_party/address_sorting/include /Ibazel-out/x64_windows-dbg/bin/external/com_github_grpc_grpc/third_party/address_sorting/include /Ibazel-out/x64_windows-dbg/genfiles/external/envoy/bazel/foreign_cc/nghttp2/include /D__CLANG_SUPPORT_DYN_ANNOTATION_ /DFMT_HEADER_ONLY /DSPDLOG_FMT_EXTERNAL /DGRPC_ARES=0 /DPB_FIELD_32BIT=1 /showIncludes /MDd /Od /Z7 /wd4117 -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -WX -Zc:__cplusplus -std:c++14 -DWIN32 -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x0602 -DNTDDI_VERSION=0x06020000 -DCARES_STATICLIB -DNGHTTP2_STATICLIB -DABSL_MALLOC_HOOK_MMAP_DISABLE -DENVOY_GOOGLE_GRPC /Fobazel-out/x64_windows-dbg/bin/source/server/_objs/server_lib/server.obj /c source/server/server.cc Execution platform: @bazel_tools//platforms:host_platform Action failed to execute: java.io.IOException: ERROR: src/main/native/windows/processes-jni.cc(429): CreateProcessWithExplicitHandles("C:\var\vcap\data\VSBuildTools\2017\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\cl.exe" /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0601 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_SECURE_NO_WARNINGS(...)): command is longer than CreateProcessW's limit (32767 characters) Target //source/exe:envoy-static failed to build INFO: Elapsed time: 318.808s, Critical Path: 146.49s INFO: 1289 processes: 1289 local. FAILED: Build did NOT complete successfully

[laszlocsomor]

@wrowe , I think you'll need 0.25.0 or newer. The 0.24.x baseline (https://github.com/bazelbuild/bazel/commit/235e76b0e756d05599a6cbe1663ff8e13df84a86) is from March 1, while compiler_param_file was added in mid-March (https://github.com/bazelbuild/bazel/commit/df3d8cb9312181d8362f7cfda3c193505bd51582).

[wrowe]

We will retest and report back after rebasing envoy to something contemporaneous to either 0.25 or 0.27, thanks!

[wrowe]

This issue is still causing us problems when building Envoy on Windows. In addition to the workaround of renaming the "_virtual_includes" directory to "v", we also find it necessary to strip off non-existent include directories from the command line. (our non-portable hack: greenhouse-org@876b199)

So we're not 100% blocked right now, but we don't know for how long our workarounds will continue to do the trick

To follow up, I suspect some form of this remove-absent-directories trick will still be needed. With so many foreign_cc targets in the project, the overabundance of -I'ncludes is going to break the 131kb cl.exe internal restriction at some point.

[achasveachas]

@laszlocsomor once we added the --features=compiler_param_file flag we were able to get much further along in the build process, so it seems to be working for now!

Are there plans to make this the default option?

[wrowe]

Are there plans to make this the default option?

Just to reiterate the open question, we believe this issue can be closed on the cc side once it is determined whether this becomes Bazel's default.

(There is an unrelated issue with respect to envoy bazel build invoking the protoc compiler during tests, with a very similar error message stream, but an unrelated solution.)

[laszlocsomor]

pinging @oquenchil about making --features=compiler_param_file the default