node_modules symlinking is incompatible with remote execution (bazel buildfarm)

[djmarcin]

🐞 bug report

Affected Rule

Any rule that invokes the link_node_modules.js script.

Is this a regression?

No

Description

The link_node_modules.js script relies on various parts of either the input source tree or runfiles tree to be writable. This is incompatible with a remote execution environment that makes these inputs readonly, such as bazel buildfarm.

🔬 Minimal Reproduction

Any build invoking link_node_modules.js when executed against a bazel-buildfarm remote execution environment.

🔥 Exception or Error

ERROR: /home/david/metawork/frontend/web/BUILD.bazel:20:11: Bundling JavaScript [vite] failed: (Exit 1): vite.sh failed: error executing command (from target //frontend/web:web)
  (cd /home/david/.cache/bazel/_bazel_david/7d1037d21a52700832f2c490bcd1c89d/execroot/metawork && \
  exec env - \
    BAZEL_NODE_MODULES_ROOTS='' \
    COMPILATION_MODE=fastbuild \
  bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/vite/bin/vite.sh build --out-dir dist --emptyOutDir --logLevel error '--bazel_node_modules_manifest=bazel-out/k8-fastbuild/bin/frontend/web/_web.module_mappings.json' '--bazel_node_working_dir=bazel-out/k8-fastbuild/bin/frontend/web')
# Configuration: 8fa33a314e8aa75f0c5bfb2c7fda3ab37bfc7fb7cbb5716938ed69ad4d4f2da8
# Execution platform: //bazel/remote:linux-x86_64
[link_node_modules.js] manifest file: bazel-out/k8-fastbuild/bin/frontend/web/_web.module_mappings.json
[link_node_modules.js] manifest contents: {
  "workspace": "metawork",
  "bin": "bazel-out/k8-fastbuild/bin",
  "roots": {
    "frontend/web": ""
  },
  "module_sets": {
    "frontend/web": {
      "vite": "bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/node_modules/vite",
      "esbuild-linux-64": "bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/node_modules/esbuild-linux-64",
      "postcss": "bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/node_modules/postcss",
      "nanoid": "bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/node_modules/nanoid",
      "picocolors": "bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/node_modules/picocolors",
      "source-map-js": "bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/node_modules/source-map-js",
      "resolve": "bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/node_modules/resolve",
      "is-core-module": "bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/node_modules/is-core-module",
      "has": "bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/node_modules/has",
      "function-bind": "bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/node_modules/function-bind",
      "path-parse": "bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/node_modules/path-parse",
      "supports-preserve-symlinks-flag": "bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/node_modules/supports-preserve-symlinks-flag",
      "rollup": "bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/node_modules/rollup"
    }
  }
}
[link_node_modules.js] startCwd: /mnt/tank/worker/shard/operations/133c1255-1109-4033-ac82-62b24b15bff2
[link_node_modules.js] execroot: /mnt/tank/worker/shard/operations/133c1255-1109-4033-ac82-62b24b15bff2
[link_node_modules.js] isExecroot: true
[link_node_modules.js] isBazelRun: false
[link_node_modules.js] creating directory bazel-out/k8-fastbuild/bin/frontend/web/node_modules in /mnt/tank/worker/shard/operations/133c1255-1109-4033-ac82-62b24b15bff2
[link_node_modules.js] creating directory frontend in /mnt/tank/worker/shard/operations/133c1255-1109-4033-ac82-62b24b15bff2
[link_node_modules.js] creating directory frontend/web in /mnt/tank/worker/shard/operations/133c1255-1109-4033-ac82-62b24b15bff2
[link_node_modules.js] creating symlink frontend/web/node_modules -> /mnt/tank/worker/shard/operations/133c1255-1109-4033-ac82-62b24b15bff2/bazel-out/k8-fastbuild/bin/frontend/web/node_modules
[link_node_modules.js] creating symlink /mnt/tank/worker/shard/operations/133c1255-1109-4033-ac82-62b24b15bff2/bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/vite/bin/vite.sh.runfiles/metawork/node_modules -> /mnt/tank/worker/shard/operations/133c1255-1109-4033-ac82-62b24b15bff2/bazel-out/k8-fastbuild/bin/frontend/web/node_modules
[link_node_modules.js] An error has been reported: [Error: EACCES: permission denied, symlink '/mnt/tank/worker/shard/operations/133c1255-1109-4033-ac82-62b24b15bff2/bazel-out/k8-fastbuild/bin/frontend/web/node_modules' -> '/mnt/tank/worker/shard/operations/133c1255-1109-4033-ac82-62b24b15bff2/bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/vite/bin/vite.sh.runfiles/metawork/node_modules'] {
  errno: -13,
  code: 'EACCES',
  syscall: 'symlink',
  path: '/mnt/tank/worker/shard/operations/133c1255-1109-4033-ac82-62b24b15bff2/bazel-out/k8-fastbuild/bin/frontend/web/node_modules',
  dest: '/mnt/tank/worker/shard/operations/133c1255-1109-4033-ac82-62b24b15bff2/bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/vite/bin/vite.sh.runfiles/metawork/node_modules'
} Error: EACCES: permission denied, symlink '/mnt/tank/worker/shard/operations/133c1255-1109-4033-ac82-62b24b15bff2/bazel-out/k8-fastbuild/bin/frontend/web/node_modules' -> '/mnt/tank/worker/shard/operations/133c1255-1109-4033-ac82-62b24b15bff2/bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/vite/bin/vite.sh.runfiles/metawork/node_modules'
Target //frontend/web:web failed to build

The directory where the symlink is being created is readonly (/mnt/tank/worker/shard/operations/133c1255-1109-4033-ac82-62b24b15bff2/bazel-out/k8-opt-exec-3AABB267/bin/external/npm_web/vite/bin/vite.sh.runfiles/metawork)

🌍 Your Environment

Operating System:

  
Linux 5.13.0-1019-aws #21~20.04.1-Ubuntu SMP Wed Mar 16 11:54:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
  

Output of bazel version:

  
Bazelisk version: v1.11.0
Build label: 6.0.0-pre.20220310.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Mar 16 18:39:34 2022 (1647455974)
Build timestamp: 1647455974
Build timestamp as int: 1647455974
  

Rules_nodejs version:

(Please check that you have matching versions between WORKSPACE file and @bazel/* npm packages.)

  
5.3.1
  

Anything else relevant?

Maybe related to https://github.com/bazelbuild/rules_nodejs/issues/1489 or https://github.com/bazelbuild/rules_nodejs/issues/824

[alexeagle]

I think we've discussed this before - we don't have much choice under node since we have to create a node_modules directory in a parent folder of the script where the require appears.

You can run the node program with --nobazel_run_linker like https://github.com/bazelbuild/rules_nodejs/blob/aa09b5796faaf2470b84246a8f00e6e27181c63b/e2e/packages/BUILD.bazel#L30 - but then vite probably won't find the node modules. Maybe --bazel_patch_module_resolver will work in it's place, as it modifies the implementation of require in node - but many nodejs programs rely on userland implementations of the resolver and those would still fail.

@gregmagolan do we have any clever ideas for the new linker?

[djmarcin]

Would setting NODE_PATH in the linker script be an option? I tried to hack it in but it seems like it fails soon after because this path is expected to exist in order to do runfiles lookups.

If these files are true runfiles, it seems like they ought to be declared as such in the node_binary rule, which it seemed like this comment was suggesting was the way forward, ~although I'm guessing the problem with that is something along the lines of "the tsc binary cannot possibly know what the npm repository rule is named in an arbitrary workspace, so it cannot be properly declared by rules_nodejs ahead of time".~ It seems like these binaries are generated by the yarn_install repository rule, so it maybe could declare the appropriate data dep.

~Although, even then it's not clear to me why the tsc binary would necessarily require the same node_modules directory as my code, but I can imagine that requirement exists via assumptions made by the tool.~

[djmarcin]

Even after hacking the system to avoid making these directories read-only on the remote worker, the action still fails soon after because of this error:

ERROR[runfiles.bash]: cannot look up runfile "npm_common/node_modules/typescript"  (RUNFILES_DIR="/mnt/tank/worker/shard/operations/7811577b-7634-4878-87ae-10fa7bd11d65/bazel-out/k8-opt-exec-3AABB267/bin/external/npm_common/typescript/bin/tsc.sh.runfiles", RUNFILES_MANIFEST_FILE="")

Dumping the files available matching typescript from the loader script reveals that locally this set of files exists:

./bazel-out/k8-opt-exec-3AABB267/bin/external/npm_common/typescript
./bazel-out/k8-opt-exec-3AABB267/bin/external/npm_common/typescript/bin/tsc.sh.runfiles/npm_common/node_modules/typescript
./bazel-out/k8-opt-exec-3AABB267/bin/external/npm_common/typescript/bin/tsc.sh.runfiles/npm_common/typescript
./bazel-out/k8-opt-exec-3AABB267/bin/external/npm_common/typescript/bin/tsc.sh.runfiles/metawork/external/npm_common/node_modules/typescript
./bazel-out/k8-opt-exec-3AABB267/bin/external/npm_common/typescript/bin/tsc.sh.runfiles/metawork/external/npm_common/typescript

But remotely only these files exist:

./bazel-out/k8-opt-exec-3AABB267/bin/external/npm_common/typescript
./bazel-out/k8-opt-exec-3AABB267/bin/external/npm_common/typescript/bin/tsc.sh.runfiles/metawork/external/npm_common/typescript
./bazel-out/k8-opt-exec-3AABB267/bin/external/npm_common/typescript/bin/tsc.sh.runfiles/npm_common/typescript

This makes me suspect there may also be implicit dependency issues with the rules. Are the rules tested/used with other remote execution systems?

[pedrobarco]

Do you have any updates on this? I'm still figuring out how to use bazel-buildfarm with a simple ts_project 😢

[alexeagle]

There has been a change since @djmarcin was here https://github.com/bazelbuild/rules_nodejs#maintenance-update so there's no one working on fixing that right now.

However, we just solved RBE in rules_js, so the next release of that (and rules_ts) should work fine with bazel-buildfarm.

@pedrobarco how big is your rules_nodejs rollout at this point? We can help you switch.

[pedrobarco]

Awesome, thanks for the update! We've been looking at it for the past couple of weeks...

I'll start by migrating a template repo I have at pedrobarco/typescript-bazel-template just to make sure we can have the same features (typescript + containers)

I'll let you know if any issue arises 👍

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 6 months. It will be closed if no further activity occurs in 30 days. Collaborators can add a "cleanup" or "need: discussion" label to keep it open indefinitely. Thanks for your contributions to rules_nodejs!