GenerateAppleSymbolsFile fails with permission error

[gferon]

I'm running a build with +output_groups=dsym and in some conditions in CI, the build fails with the following error:

Failed to write bazel-out/applebin_ios-ios_arm64-opt-ST-55c05c3eeee9/bin/XXXXXXXXXXXX.framework-intermediates/symbols_output/0D911365-E3B1-37ED-AFC6-D4FBB0160CB1.symbols: You don’t have permission to save the file “0D911365-E3B1-37ED-AFC6-D4FBB0160CB1.symbols” in the folder “symbols_output”.

When looking at the directory, it is indeed read-only but I'm failing to understand why actions.declare_directory from https://github.com/bazelbuild/rules_apple/blob/96fa885dd8310ddc65289e590f31d1f3b25f46d9/apple/internal/intermediates.bzl#L43 fails to give the rule a directory that can be written to.

This happens when running the same build multiple times, I'll try to find a sure way to reproduce it, but would be happy to get any insight in the meantime if you have any.

[thii]

Are you building with local spawn strategy? It's because Bazel adds the read-only bit to all outputs after finishing the action.

[gferon]

No, I double checked to be sure but I'm using the regular sandboxed spawn strategy.

[gferon]

FWIW, the only difference I can see between CI runners that are affected and the ones that aren't is that the --output_user_root is pointing to a symlink, pointing to another mount-point.

EDIT: I'm using Bazel 6.3.0rc1 and was seeing the same issue with 6.2.0

[thii]

I'm not sure if it's related but you can try to debug the state of the output directory by adding your debug code here:

https://github.com/bazelbuild/rules_apple/blob/96fa885dd8310ddc65289e590f31d1f3b25f46d9/apple/internal/partials/apple_symbols_file.bzl#L78-L80

[gferon]

Progress report on the issue: disabling the sandbox seems to make the issue go away, but I need to wait to confirm that. It's definitely a sort of Heisenbug.

EDIT: --strategy=GenerateAppleSymbolsFile=local nearly confirmed to fix our issue after repeated builds in CI.

EDIT2: Would adding an execution_requirements with no-sandbox be an option here?

[thii]

Would adding an execution_requirements with no-sandbox be an option here?

I wouldn't recommend doing so until we understand the issue here.

you can try to debug the state of the output directory

Have you tried this? When it failed, did the directory exist before the execution? If yes, what were its attributes?

[thii]

I noticed this in your log: XXXXXXXXXXXX.framework-intermediates, but include_symbols_in_bundle only exists in {i,mac}os_application. Are you building with a non-standard rule?

[gferon]

The application only depends on a bunch of ios_framework and apple_xcframework (that have been imported). I'll try to figure out why the symbols are being generated for one of the xcframeworks.

[nataliejameson]

Hiya! We ran into this problem too, and I think I've tracked down what the actual issue is.

Just to give an overview of our setup, we have a volume that we store our bazel output and sandboxes on. Something like this:

startup --output_user_root=/Volumes/somelabel
startup --output_base=/Volumes/somelabel/bazel_output
build --sandbox_base=/Volumes/somelabel/bazel_sandbox

When we would do a build, we'd get errors that looked something like this:

Failed to write bazel-out/applebin_ios-ios_arm64-opt-ST-40686511a296/bin/PACKAGE/BIN_NAME-intermediates/symbols_output/319AA7FB-686D-3E56-BA05-55B83A76C1C1.symbols: You don’t have permission to save the file “319AA7FB-686D-3E56-BA05-55B83A76C1C1.symbols” in the folder “symbols_output”.

However, manually running that sb profile that bazel used we could do things like touching files in that symbols_output dir. I kind of ran around thinking maybe it was accessing a dir incorrectly, or breaking out of the sandbox and going cross fs, or something else weird. Nope, nothing like that: It seems like xcrun symbols was trying to write to /Volumes/somelabel/.TemporaryItems, then move those into place afterwards. The error message was just super confusing. I'm guessing this isn't hit very often because it looks like you have to be running your sandbox on a non-root volume.

So, two things you can do in your bazelrc that reliabily fixed it for us:

build --sandbox_writable_path=/Volumes/somelabel/.TemporaryItems

or

build --strategy=GenerateAppleSymbolsFile=local

This took a bit to track down, but if you watch the Console while your task fails and can filter out all the junk, you'll see a message like this:

Sandbox: symbols(78835) deny(1) file-write-create /Volumes/somelabel/.TemporaryItems/folders.503/TemporaryItems/NSIRD_SamplingTools_eTcLhh

showing what was denied. Hope that helps anyone else who stumbles into this error, it's kinda niche 😅

[brentleyjones]

build --strategy=GenerateAppleSymbolsFile=local

Should we make the action non-sandboxed by default?

[gferon]

build --strategy=GenerateAppleSymbolsFile=local

We actually tried this for quite a while and it didn't work. I'll try the advice from @nataliejameson and see if we can re-enable the sandbox on macOS CI runners.