bazel
bazel copied to clipboard
bazelbuild
Execution root symlink forest includes all files in a package instead of only ones needed
Description of the bug:
When running a build, the symlink forest will include all files from a package beyond files actually needed, used, or referenced by a build.
This leads to various quality-of-life, and reproducibility issues.
An example of the output base from the open source rules_xcodeproj looks like this when a target in the root package is built:
Which category does this issue belong to?
No response
What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
- Clone rules_xcodeproj
bazel build tools:xcodeprojcdintobazel-output-base/execroot/_main/- Notice the execroot has symlinked two packages used in that build
cdback to the workspace rootbazel build buildifier.check- Notice the execroot has symlnked all files in the root, including BUILD files and
.gitdirectories
- Notice the execroot has symlnked all files in the root, including BUILD files and
Which operating system are you running Bazel on?
macOS 14.0
What is the output of bazel info release?
release 6.4.0rc1
If bazel info release returns development version or (@non-git), tell us how you built Bazel.
No response
What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?
No response
Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.
No
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
This has existed for a while as a nuisance (if you traverse bazel-out), it almost always tricks tools into thinking it's a Git directory, but became a more acute problem in the iOS community with Xcode 15 as the IDE performance is extremely degraded trying to parse what Git views as an extremely large Git diff in the output base.
bazelbuild.slack.com discussion
This appears to reproduce in all scenarios in Bazel 7 (tested on rc3).
In other words, both bazel build tools:xcodeproj and bazel build buildifier.check in the repro steps above result in all files and directories in the repository being symlinked.
This is unfortunately a known limitation of skymeld (https://github.com/bazelbuild/bazel/issues/14057), which is on by default in Bazel 7. In this mode, we create a symlink to each entry under the source root.
If this breaks you, please include --noexperimental_merged_skyframe_analysis_execution in your blazerc to disable skymeld.
If I understand correctly, the main nuisance here is with the .git file also being symlinked to from the execroot, right?
If I understand correctly, the main nuisance here is with the .git file also being symlinked to from the execroot, right?
Yes. If minimally that wasn't symlinked, we would be fine. One step more would to also not symlink anything in .bazelignore. Ideally only files that need to be symlinked are symlinked.
If this breaks you, please include
--noexperimental_merged_skyframe_analysis_executionin your blazerc to disable skymeld.
Sadly I will have to include that by default for rules_xcodeproj if this ships this way. It breaks Xcode as is.
@joeleba Should we @bazel-io flag this for Bazel 7?
Interestingly Bazel itself depends on source files in .git at https://github.com/bazelbuild/bazel/blob/07a571f70beed3fc46c14b4c933db91e79cfdb80/BUILD#L54-L61. The minimal solution of not symlinking just .git may thus not be sufficiently configurable. Going by .bazelignore should cover this use case though.
The commit that tells bazel to skip the paths listed in .bazelignore was cherrypicked into the release. You should be able to use that as a workaround. I'll lower the priority of this bug.
@joeleba Thanks! I assume that only applies to when using Skymeld? If we don't use Skymeld, .bazelignore won't fix the originally reported issue?
That's true at least for Skymeld: the symlink planting should repect .bazelignore. I wasn't aware that this was an issue without Skymeld. AFAIK the noskymeld case actually goes through all the packages and only plants symlinks for the necessary ones.
AFAIK the noskymeld case actually goes through all the packages and only plants symlinks for the necessary ones
Sure, but the issue is when a target in the root package is built, it will symlink the .git directory, even though nothing has a dependency on that directory.
Just confirmed that this is an issue for the noskymeld case. We had to disable Skymeld for a 7.2.0rc1 related bug and we immediately hit this issue.