`bazel clean` always required after failed `configure_make`

[denglesberg-splunk]

I've reproduced this to a minimal reproduction, attached below. The issue I observe is that when attempting to build again after a failed build, there's something wrong with the cache / workspace and the build fails immediately. See below - the first build is expected to fail and appears to be working as expected. The 2nd build is where the issue lies:

$ bazel build :main
INFO: Analyzed target //:main (43 packages loaded, 757 targets configured).
INFO: Found 1 target...
ERROR: /home/denglesberg/git/bazel_experiment/main/BUILD.bazel:8:15: Foreign Cc - Configure: Building main failed: (Exit 127): bash failed: error executing command (from target //:main) /bin/bash -c bazel-out/k8-fastbuild/bin/main_foreign_cc/wrapper_build_script.sh

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
rules_foreign_cc: Build failed!
rules_foreign_cc: Keeping temp build directory and dependencies directory for debug.
rules_foreign_cc: Please note that the directories inside a sandbox are still cleaned unless you specify --sandbox_debug Bazel command line flag.
rules_foreign_cc: Printing build logs:
_____ BEGIN BUILD LOGS _____

Bazel external C/C++ Rules. Building library main

Environment:______________
BUILD_SCRIPT=bazel-out/k8-fastbuild/bin/main_foreign_cc/build_script.sh
EXT_BUILD_ROOT=/home/denglesberg/.cache/bazel/_bazel_denglesberg/0e18f310661938cc3e7daef810074c88/sandbox/linux-sandbox/10/execroot/__main__
BUILD_LOG=bazel-out/k8-fastbuild/bin/main_foreign_cc/Configure.log
PWD=/home/denglesberg/.cache/bazel/_bazel_denglesberg/0e18f310661938cc3e7daef810074c88/sandbox/linux-sandbox/10/execroot/__main__
BUILD_WRAPPER_SCRIPT=bazel-out/k8-fastbuild/bin/main_foreign_cc/wrapper_build_script.sh
TMPDIR=/tmp
EXT_BUILD_DEPS=/home/denglesberg/.cache/bazel/_bazel_denglesberg/0e18f310661938cc3e7daef810074c88/sandbox/linux-sandbox/10/execroot/__main__/bazel-out/k8-fastbuild/bin/main.ext_build_deps
BUILD_TMPDIR=/home/denglesberg/.cache/bazel/_bazel_denglesberg/0e18f310661938cc3e7daef810074c88/sandbox/linux-sandbox/10/execroot/__main__/bazel-out/k8-fastbuild/bin/main.build_tmpdir
SHLVL=2
INSTALLDIR=/home/denglesberg/.cache/bazel/_bazel_denglesberg/0e18f310661938cc3e7daef810074c88/sandbox/linux-sandbox/10/execroot/__main__/bazel-out/k8-fastbuild/bin/main
PATH=/home/denglesberg/.cache/bazel/_bazel_denglesberg/0e18f310661938cc3e7daef810074c88/sandbox/linux-sandbox/10/execroot/__main__:/home/denglesberg/.cache/bazelisk/downloads/bazelbuild/bazel-6.1.1-linux-x86_64/bin:/home/denglesberg/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
_=/usr/bin/env
__________________________
Can not copy /home/denglesberg/.cache/bazel/_bazel_denglesberg/0e18f310661938cc3e7daef810074c88/sandbox/linux-sandbox/10/execroot/__main__/pkg-config
+ mkdir -p /home/denglesberg/.cache/bazel/_bazel_denglesberg/0e18f310661938cc3e7daef810074c88/sandbox/linux-sandbox/10/execroot/__main__/bazel-out/k8-fastbuild/bin/main.build_tmpdir/main
+ ARFLAGS=rcsD
+ AR_FLAGS=rcsD
+ ASFLAGS='-U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__=redacted -D__TIMESTAMP__=redacted -D__TIME__=redacted'
+ CFLAGS='-U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__=redacted -D__TIMESTAMP__=redacted -D__TIME__=redacted'
+ CXXFLAGS='-U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -std=c++0x -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__=redacted -D__TIMESTAMP__=redacted -D__TIME__=redacted'
+ LDFLAGS='-fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lstdc++ -lm'
+ AR=/usr/bin/ar
+ CC=/usr/bin/gcc
+ CXX=/usr/bin/gcc
+ RANLIB=:
+ CPPFLAGS=
+ /home/denglesberg/.cache/bazel/_bazel_denglesberg/0e18f310661938cc3e7daef810074c88/sandbox/linux-sandbox/10/execroot/__main__/./configure --prefix=/home/denglesberg/.cache/bazel/_bazel_denglesberg/0e18f310661938cc3e7daef810074c88/sandbox/linux-sandbox/10/execroot/__main__/bazel-out/k8-fastbuild/bin/main.build_tmpdir/main
/home/denglesberg/.cache/bazel/_bazel_denglesberg/0e18f310661938cc3e7daef810074c88/sandbox/linux-sandbox/10/execroot/__main__/./configure: line 1: bin/does_not_exist.sh: No such file or directory
_____ END BUILD LOGS _____
rules_foreign_cc: Build wrapper script location: bazel-out/k8-fastbuild/bin/main_foreign_cc/wrapper_build_script.sh
rules_foreign_cc: Build script location: bazel-out/k8-fastbuild/bin/main_foreign_cc/build_script.sh
rules_foreign_cc: Build log location: bazel-out/k8-fastbuild/bin/main_foreign_cc/Configure.log

Target //:main failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 11.515s, Critical Path: 11.23s
INFO: 7 processes: 6 internal, 1 linux-sandbox.
FAILED: Build did NOT complete successfully

$ bazel build :main
INFO: Analyzed target //:main (1 packages loaded, 58 targets configured).
INFO: Found 1 target...
ERROR: /home/denglesberg/git/bazel_experiment/main/BUILD.bazel:8:15: Foreign Cc - Configure: Building main failed: missing input file '//:bazel-out/_tmp/actions/stderr-6'
ERROR: /home/denglesberg/git/bazel_experiment/main/BUILD.bazel:8:15: Foreign Cc - Configure: Building main failed: missing input file '//:bazel-out/_tmp/actions/stderr-7'
ERROR: /home/denglesberg/git/bazel_experiment/main/BUILD.bazel:8:15: Foreign Cc - Configure: Building main failed: missing input file '//:bazel-out/_tmp/actions/stdout-6'
ERROR: /home/denglesberg/git/bazel_experiment/main/BUILD.bazel:8:15: Foreign Cc - Configure: Building main failed: missing input file '//:bazel-out/_tmp/actions/stdout-7'
[reproduction.tar.gz](https://github.com/bazelbuild/rules_foreign_cc/files/11122430/reproduction.tar.gz)

ERROR: /home/denglesberg/git/bazel_experiment/main/BUILD.bazel:8:15: Foreign Cc - Configure: Building main failed: 4 input file(s) do not exist
Target //:main failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /home/denglesberg/git/bazel_experiment/main/BUILD.bazel:8:15 Foreign Cc - Configure: Building main failed: 4 input file(s) do not exist
INFO: Elapsed time: 0.072s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
FAILED: Build did NOT complete successfully

This is fixed by running bazel clean, after which bazel build :main again works as expected

In my experience using bazel, typically one does not have to clean so often, so I suspect that there is some issue with the configure_make() rule.

The minimal reproduction is attached.

[denglesberg-splunk]

reproduction.tar.gz

[alexeagle]

Ping @jsharpe ? Sounds like there's just an input to the repository rule which isn't referenced by a label during the rule execution, so bazel doesn't know that changes to it should invalidate the repo rule?

[jsharpe]

I don't have much bandwidth available to look at this but I'd suspect the most likely candidate is that the configure_make rule is modifying one of the input files as part of the build? Although aren't the files like //:bazel-out/_tmp/actions/stdout-7 made by bazel itself; seems like bazel might be leaving some stale ones around in the tree which are possibly being picked up by the input file glob? Those are my guesses without looking at this in any more detail.

[denglesberg-splunk]

In the minimal reproduction, there are three files: WORKSPACE.bazel BUILD.bazel and configure. None of them appear to be modified by the bazel execution.

The input file rule is:

filegroup(
    name = "source",
    srcs = glob(["**"], exclude=["bazel*"]),
)

which I would expect to exclude the bazel-created directories from the :source target. If that's not happening though that could explain the problem here.

[alexeagle]

Indeed I think you found the problem, exclude=["bazel*/**"] makes the glob exclude the files correctly

[denglesberg-splunk]

Thanks @alexeagle for the help - indeed the problem is with the glob exclude - it needs to exclude bazel*/** to actually exclude the bazel symlinks.

I'll close this.