Argument list too long error

[egolearner]

Envirment Info

Bazel plugin version: v2019.07.23.0.3 bazel version: 0.24.1 OS version: mac 10.14.4 Clion version: Clion 2019.1.4

Error message

Error:Cannot run program "/usr/local/bin/bazel" (in directory "project xxx"): error=7, Argument list too long

Bazel Console message

Syncing project: Sync (incremental)...
Updating VCS...
Running Bazel info...
Command: /usr/local/bin/bazel info --tool_tag=ijwb:CLion --curses=no --color=yes --progress_in_terminal_title=no --

Computing VCS working set...
Querying targets in project directories...
Command: /usr/local/bin/bazel query --tool_tag=ijwb:CLion --output=label_kind --keep_going attr("tags", "^((?!manual).)*$", //...:all - //bazel-bin/...:all - //bazel-genfiles/...:all - //bazel-out/...:all - //bazel-testlogs/...:all - //bazel-tensorflow/...:all - //.clwb/...:all) --

Sync targets from project view directories:
  //third_party/toolchains/preconfig/win_1803/py36:python_headers
  //third_party/toolchains/preconfig/win_1803/py36:numpy_headers
  //third_party/toolchains/preconfig/win_1803/bazel_018:stl
  //third_party/toolchains/preconfig/win_1803/bazel_018:malloc
  //third_party/toolchains/preconfig/ubuntu16.04/tensorrt5.1:tensorrt
  //third_party/toolchains/preconfig/ubuntu16.04/tensorrt5.1:tensorrt_headers
  //third_party/toolchains/preconfig/ubuntu16.04/tensorrt5:tensorrt
  //third_party/toolchains/preconfig/ubuntu16.04/tensorrt5:tensorrt_headers
  //third_party/toolchains/preconfig/ubuntu16.04/py3:python_headers
  //third_party/toolchains/preconfig/ubuntu16.04/py3:numpy_headers
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:libdevice_root
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cusparse
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cusolver
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cupti_headers
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cupti_headers_virtual
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cupti_dsos
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cudnn_header
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cudart_static
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cuda_driver
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cuda
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:curand
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cufft
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cudnn
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cudart
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cublas_headers
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cublas_headers_virtual
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cuda_headers
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cuda_headers_virtual
  //third_party/toolchains/preconfig/ubuntu16.04/cuda10.0-cudnn7/cuda:cublas
  //third_party/toolchains/preconfig/ubuntu16.04/clang:malloc
  //third_party/toolchains/preconfig/ubuntu14.04/tensorrt5:tensorrt
  //third_party/toolchains/preconfig/ubuntu14.04/tensorrt5:tensorrt_headers
  //third_party/toolchains/preconfig/ubuntu14.04/py3:python_headers
  //third_party/toolchains/preconfig/ubuntu14.04/py3:numpy_headers
  //third_party/toolchains/preconfig/ubuntu14.04/cuda10.0-cudnn7/cuda:libdevice_root
  //third_party/toolchains/preconfig/ubuntu14.04/cuda10.0-cudnn7/cuda:cusparse
  //third_party/toolchains/preconfig/ubuntu14.04/cuda10.0-cudnn7/cuda:cusolver
  //third_party/toolchains/preconfig/ubuntu14.04/cuda10.0-cudnn7/cuda:cupti_headers
  //third_party/toolchains/preconfig/ubuntu14.04/cuda10.0-cudnn7/cuda:cupti_headers_virtual
  //third_party/toolchains/preconfig/ubuntu14.04/cuda10.0-cudnn7/cuda:cupti_dsos
  //third_party/toolchains/preconfig/ubuntu14.04/cuda10.0-cudnn7/cuda:cudnn_header
  //third_party/toolchains/preconfig/ubuntu14.04/cuda10.0-cudnn7/cuda:cudart_static
  //third_party/toolchains/preconfig/ubuntu14.04/cuda10.0-cudnn7/cuda:cuda_driver
  //third_party/toolchains/preconfig/ubuntu14.04/cuda10.0-cudnn7/cuda:cuda
  //third_party/toolchains/preconfig/ubuntu14.04/cuda10.0-cudnn7/cuda:curand
  //third_party/toolchains/preconfig/ubuntu14.04/cuda10.0-cudnn7/cuda:cufft
  //third_party/toolchains/preconfig/ubuntu14.04/cuda10.0-cudnn7/cuda:cudnn
  //third_party/toolchains/preconfig/ubuntu14.04/cuda10.0-cudnn7/cuda:cudart
  //third_party/toolchains/preconfig/ubuntu14.04/cuda10.0-cudnn7/cuda:cublas_headers
  //third_party/toolchains/preconfig/ubuntu14.04/cuda10.0-cudnn7/cuda:cublas_headers_virtual

Plus 7801 more targets
Building blaze targets...
Command: /usr/local/bin/bazel build --tool_tag=ijwb:CLion --keep_going --build_event_binary_file=/var/folders/26/1y98zzm50l90ss8h_plr3nvm0000gn/T/intellij-bep-69d84afb-4868-4ef3-8580-a6e97d0a08d2 --nobuild_event_binary_file_path_conversion --curses=no --color=yes --progress_in_terminal_title=no --aspects=@intellij_aspect//:intellij_info_bundled.bzl%intellij_info_aspect --override_repository=intellij_aspect=/Users/xxx/Library/Application Support/JetBrains/Toolbox/apps/CLion/ch-0/191.7479.33/CLion.app.plugins/clwb/aspect --output_groups=intellij-info-cpp,intellij-info-generic,intellij-info-py,intellij-resolve-cpp,intellij-resolve-py -- //third_party/toolchains/preconfig/win_1803/py36:python_headers //third_party/toolchains/preconfig/win_1803/py36:numpy_headers //third_party/toolchains/preconfig/win_1803/bazel_018:stl //third_party/toolchains/preconfig/win_1803/bazel_018:malloc //third_party/toolchains/preconfig/ubuntu16.04/tensorrt5.1:tensorrt //third_party/toolchains/preconfi...
build invocation result: fatal_error
Sync finished

Timing summary:
BlazeInvocation: 4.1s, Prefetching: 0ms, Other: 0ms
Sync failed

[jin]

For a workaround, can you try setting derive_targets_from_directories: false in your project view file, and manually set the targets: you want to sync? e.g.

derive_targets_from_directories: false

targets:
   //my:target
   //foo/package/...

[egolearner]

Thanks @jin

[jin]

This is an issue with macOS with a much lower ARG_MAX value. Please use the workaround in https://github.com/bazelbuild/intellij/issues/1081#issuecomment-520830345 if you run into this issue.

[MikhailTymchukFT]

Is it possible to pass arguments via file to avoid this limit? cat /path/to/file | xargs ./exec https://apple.stackexchange.com/a/254019

[brendandouglas]

Good idea. At the moment we're automatically splitting targets across multiple builds when it looks like we'd exceed ARG_MAX. That won't affect performance too much given the level of caching in the bazel server.

It's definitely a hack though -- if we see users run into issues with that approach we'll take a look at your suggestion.

[storypku]

For a workaround, can you try setting derive_targets_from_directories: false in your project view file, and manually set the targets: you want to sync? e.g.

derive_targets_from_directories: false

targets:
   //my:target
   //foo/package/...

Hi @jin , I run into the same "Argument list too long" error when running something similar to the following

bazel build $(bazel query //A/... union //B... except //B/C/... except //B/D/...

And it seems that it can be bypassed by using the following alternative:

bazel build -- //A/... //B/... -//B/C/... -//B/D/...

My machine info:

Bazel version: 3.4.1 on x86_64 
Ubuntu 18.04.02

Here is my question: which one is better practice for such multiple targets build task ?

[Elentary]

Good idea. At the moment we're automatically splitting targets across multiple builds when it looks like we'd exceed ARG_MAX. That won't affect performance too much given the level of caching in the bazel server.

It's definitely a hack though -- if we see users run into issues with that approach we'll take a look at your suggestion.

Do you ? I have encountered this problem on a fresh version of the plugin in CLion just this day

[Elentary]

@brendandouglas

[Elentary]

@jin @brendandouglas @AlexeyGy The problem still exists in current plugin CLion version and is a critical blocker for my team and me. Are you sure the splitting works ? I am attaching my ARG_MAX settings just to be sure it's okay.

[ amareelez@amareelez-desknix <NV> outdated ~/sdc ]$ grep ARG_MAX /usr/include/linux/limits.h
#define ARG_MAX       131072    /* # bytes of args + environ for exec() */
[ amareelez@amareelez-desknix <NV> outdated ~/sdc ]$ getconf ARG_MAX
2097152
[ amareelez@amareelez-desknix <NV> outdated ~/sdc ]$ grep ARG /usr/include/linux/binfmts.h
 * execve() system call.  MAX_ARG_STRLEN is essentially random but serves to
 * MAX_ARG_STRINGS is chosen to fit in a signed 32-bit integer.
#define MAX_ARG_STRLEN (PAGE_SIZE * 32)
#define MAX_ARG_STRINGS 0x7FFFFFFF
[ amareelez@amareelez-desknix <NV> outdated ~/sdc ]$ getconf PAGE_SIZE
4096

[Elentary]

@jin @brendandouglas @AlexeyGy

Issue still persists :(