How to call a nodejs_binary in genrule

[aherrmann]

🐞 bug report

Affected Rule

The issue is caused by the rule: nodejs_binary and genrule

Is this a regression?

Not that I'm aware.

Description

Trying to execute a nodejs_binary in a genrule yields errors due to missing node modules.

🔬 Minimal Reproduction

Create the following files:

BUILD.bazel:

load("@npm//grunt-cli:index.bzl", "grunt")

grunt(
    name = "grunt",
    data = [
        "@npm//grunt",
    ],
)

genrule(
    name = "help",
    cmd = """\
$(execpath :grunt) --version >$(OUTS)
$(execpath :grunt) --help >>$(OUTS)
    """,
    outs = ["help.txt"],
    tools = [
        ":grunt",
    ],
)

package.json:

{
    "dependencies": {
        "grunt": "^1.0.0"
    }
}

WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
    name = "build_bazel_rules_nodejs",
    sha256 = "c97bf38546c220fa250ff2cc052c1a9eac977c662c1fc23eda797b0ce8e70a43",
    urls = ["https://github.com/bazelbuild/rules_nodejs/releases/download/1.1.0/rules_nodejs-1.1.0.tar.gz"],
)
load("@build_bazel_rules_nodejs//:index.bzl", "node_repositories")
node_repositories(package_json = ["//:package.json"])
load("@build_bazel_rules_nodejs//:index.bzl", "yarn_install")
yarn_install(
    name = "npm",
    package_json = "//:package.json",
    yarn_lock = "//:yarn.lock",
)

Execute the following commands:

yarn install
bazel build --define=VERBOSE_LOGS=1 //:help && cat bazel-bin/help.txt

🔥 Exception or Error

ERROR: .../nodejs-binary-repro/BUILD.bazel:20:1: Executing genrule //:help failed (Exit 1) bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
[link_node_modules.js] module manifest: workspace __main__, bin bazel-out/host/bin, root npm/node_modules with first-party packages
 { __main__: [ 'bin', '__main__' ] }
[link_node_modules.js] resolved root npm/node_modules to ../npm/node_modules
[link_node_modules.js] cwd .../12a58783da5a0c493829d8d7084201c0/sandbox/linux-sandbox/6/execroot/__main__
[link_node_modules.js] symlink( node_modules -> ../npm/node_modules )
[link_node_modules.js] Error: ENOENT: no such file or directory, chdir '.../12a58783da5a0c493829d8d7084201c0/sandbox/linux-sandbox/6/execroot/__main__' -> '../npm/node_modules'
    at process.chdir (internal/process/main_thread_only.js:29:13)
    at .../12a58783da5a0c493829d8d7084201c0/external/build_bazel_rules_nodejs/internal/linker/index.js:448:21
    at Generator.next ()
    at fulfilled (.../12a58783da5a0c493829d8d7084201c0/external/build_bazel_rules_nodejs/internal/linker/index.js:3:58) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'chdir',
  path: '.../12a58783da5a0c493829d8d7084201c0/sandbox/linux-sandbox/6/execroot/__main__',
  dest: '../npm/node_modules'
}
Target //:help failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1.291s, Critical Path: 0.30s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

Expected Output

The build should succeed and the generated file should contain the grunt-cli and grunt version numbers and grunt's help message.

...
INFO: Build completed successfully, 2 total actions
grunt-cli v1.2.0
grunt v1.0.4
Grunt: The JavaScript Task Runner (v1.0.4)

Usage
 grunt [options] [task [task ...]]

Options
    --help, -h  Display this help text.
    --base, -b  Specify an alternate base path. By default, all file paths are
                relative to the Gruntfile. (grunt.file.setBase) *
    --no-color  Disable colored output.
   --gruntfile  Specify an alternate Gruntfile. By default, grunt looks in the
                current or parent directories for the nearest Gruntfile.js or
                Gruntfile.coffee file.
   --debug, -d  Enable debugging mode for tasks that support it.
       --stack  Print a stack trace when exiting with a warning or fatal error.
   --force, -f  A way to force your way past warnings. Want a suggestion? Don't
                use this option, fix your code.
       --tasks  Additional directory paths to scan for task and "extra" files.
                (grunt.loadTasks) *
         --npm  Npm-installed grunt plugins to scan for task and "extra" files.
                (grunt.loadNpmTasks) *
    --no-write  Disable writing files (dry run).
 --verbose, -v  Verbose mode. A lot more information output.
 --version, -V  Print the grunt version. Combine with --verbose for more info.
  --completion  Output shell auto-completion rules. See the grunt-cli
                documentation for more information.
     --require  Specify a language interpreter to require first if you are
                writing your Gruntfile in a language Grunt doesn't support by
                default.

Options marked with * have methods exposed via the grunt API and should instead
be specified inside the Gruntfile wherever possible.

Available tasks
(no tasks found)

The list of available tasks may change based on tasks directories or grunt
plugins specified in the Gruntfile or via command-line options.

For more information, see http://gruntjs.com/

Known Workaround

Link runfiles node_modules

Adding the following to the genrule's tools attribute:

"@npm//grunt-cli",

And adding the following as the first line in the genrule's command:

ln -s $(execpath :grunt).runfiles/npm/node_modules .

With these changes the build does not fail and the generated file contains the expected output.

However, this seems like a hack and wouldn't work on Windows, where symbolic links are not generally available.

Adding all node module dependencies

Adding the following to the genrule's tools attribute:

"@npm//abbrev",
"@npm//ansi-styles",
"@npm//async",
"@npm//balanced-match",
"@npm//brace-expansion",
"@npm//chalk",
"@npm//coffeescript",
"@npm//color-convert",
"@npm//color-name",
"@npm//colors",
"@npm//concat-map",
"@npm//dateformat",
"@npm//escape-string-regexp",
"@npm//eventemitter2",
"@npm//exit",
"@npm//findup-sync",
"@npm//fs.realpath",
"@npm//getobject",
"@npm//glob",
"@npm//grunt",
"@npm//grunt-cli",
"@npm//grunt-known-options",
"@npm//grunt-legacy-log",
"@npm//grunt-legacy-log-utils",
"@npm//grunt-legacy-util",
"@npm//has-flag",
"@npm//hooker",
"@npm//iconv-lite",
"@npm//inflight",
"@npm//inherits",
"@npm//isexe",
"@npm//js-yaml",
"@npm//lodash",
"@npm//minimatch",
"@npm//mkdirp",
"@npm//nopt",
"@npm//once",
"@npm//path-is-absolute",
"@npm//resolve",
"@npm//rimraf",
"@npm//safer-buffer",
"@npm//sprintf-js",
"@npm//supports-color",
"@npm//underscore.string",
"@npm//util-deprecate",
"@npm//which",
"@npm//wrappy",

This is an unsatisfactory solution as one has to seemingly manually trace all out transitive runtime dependencies and list them as tools dependencies to the genrule.

🌍 Your Environment

Operating System:

  
openSUSE Tumbleweed VERSION 20190529
  

Output of bazel version:

  
Build label: 1.1.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Oct 21 08:44:00 2019 (1571647440)
Build timestamp: 1571647440
Build timestamp as int: 1571647440
  

Rules_nodejs version:

  
1.1.0
  

Anything else relevant?

https://github.com/bazelbuild/rules_nodejs/issues/1451 asks about calling a nodejs_binary in a sh_binary and could be related. However, the discussion there seems to focus on general runfiles (data dependencies), whereas this question is explicitly concerned with node module dependencies.

[bshepherdson]

I ran into this too, while trying to pass a nodejs_binary as an extra binary to a Bazel run action. The main binary in my case is protoc, and the nodejs_binary that's breaking is the TypeScript plugin, protoc-gen-ts.

I can bazel run the nodejs_binary just fine, but when I try to use it as part of the tools, I get:

ERROR: missing input file 'external/npm/node_modules/create-frame/node_modules/define-pro
perty/LICENSE', owner: '@npm//:node_modules/create-frame/node_modules/define-property/LIC
ENSE'
ERROR: missing input file 'external/npm/node_modules/create-frame/node_modules/define-pro
perty/README.md', owner: '@npm//:node_modules/create-frame/node_modules/define-property/R
EADME.md'
ERROR: missing input file 'external/npm/node_modules/create-frame/node_modules/define-pro
perty/index.js', owner: '@npm//:node_modules/create-frame/node_modules/define-property/in
dex.js'
ERROR: missing input file 'external/npm/node_modules/create-frame/node_modules/define-pro
perty/package.json', owner: '@npm//:node_modules/create-frame/node_modules/define-propert
y/package.json'

... and dozens more

I'm currently trying to figure out the right things to include as inputs or tools so it can actually find the right files, but no luck so far. Since I'm working with actions, it's a bit tricky to do the symlink workaround above, but I'll try that too.

[bshepherdson]

Hm, my problem above seems to be unrelated in the end. Missing input files like that seem to be related to https://github.com/bazelbuild/bazel/issues/5437 and not part of me doing anything wrong. After a bazel clean --expunge everything was suddenly working.

[mattem]

This is due to the generated nodejs_binary having the --nobazel_patch_module_resolver flag set.

Can workaround by either defining your own nodejs_binary or npm_package_bin, depending on the use here, or supplying outs or output_dir = 1 to the generate marco, which will cause it to generate an npm_package_bin instead.

Alternatively, supply the args directly to the generated grunt macro:

grunt(
    name = "grunt",
    output_dir = 1,
    data = ["@npm//grunt"],
    args = ["--help"]
)

Produces:

bazel build //:grunt

INFO: Analyzed target //:grunt (214 packages loaded, 3354 targets configured).
INFO: Found 1 target...
INFO: From Action grunt:
Grunt: The JavaScript Task Runner (v1.1.0)

Usage
 grunt [options] [task [task ...]]

Options
    --help, -h  Display this help text.
    --base, -b  Specify an alternate base path. By default, all file paths are
                relative to the Gruntfile. (grunt.file.setBase) *
    --no-color  Disable colored output.
   --gruntfile  Specify an alternate Gruntfile. By default, grunt looks in the
                current or parent directories for the nearest Gruntfile.js or
                Gruntfile.coffee file.
   --debug, -d  Enable debugging mode for tasks that support it.
       --stack  Print a stack trace when exiting with a warning or fatal error.
   --force, -f  A way to force your way past warnings. Want a suggestion? Don't
                use this option, fix your code.
       --tasks  Additional directory paths to scan for task and "extra" files.
                (grunt.loadTasks) *
         --npm  Npm-installed grunt plugins to scan for task and "extra" files.
                (grunt.loadNpmTasks) *
    --no-write  Disable writing files (dry run).
 --verbose, -v  Verbose mode. A lot more information output.
 --version, -V  Print the grunt version. Combine with --verbose for more info.
  --completion  Output shell auto-completion rules. See the grunt-cli
                documentation for more information.
     --require  Specify a language interpreter to require first if you are
                writing your Gruntfile in a language Grunt doesn't support by
                default.

Options marked with * have methods exposed via the grunt API and should instead
be specified inside the Gruntfile wherever possible.

Available tasks
(no tasks found)

The list of available tasks may change based on tasks directories or grunt
plugins specified in the Gruntfile or via command-line options.

For more information, see http://gruntjs.com/
Target //:grunt up-to-date:
  bazel-bin/grunt
INFO: Elapsed time: 2.675s, Critical Path: 1.51s
INFO: 1 process: 1 darwin-sandbox.
INFO: Build completed successfully, 9 total actions

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs in two weeks. Collaborators can add a "cleanup" or "need: discussion" label to keep it open indefinitely. Thanks for your contributions to rules_nodejs!

[github-actions[bot]]

This issue was automatically closed because it went two weeks without a reply since it was labeled "Can Close?"

[enigmatix]

Just wanted to +1 this, I ran into the issue today and took me quite a while to understand before knowing what to search and find the workaround here.

[christianscott]

Future me: run_node is what you want.

• Docs: https://bazelbuild.github.io/rules_nodejs/Providers.html#run_node
• Example: https://github.com/bazelbuild/rules_nodejs/blob/stable/packages/typescript/test/some_module/run_node_test.bzl#L6

[alexeagle]

note that our long-term plan is a rewrite of nodejs_binary that exclusively uses runfiles as the "static linker" so that run_node can go away. proof of concept: https://github.com/aspect-build/rules_js/blob/main/example/BUILD.bazel

[brian-tecton-ai]

Link above for run_node seems broken -- is there an up-to-date example of the "canonical" workaround? One thing to note is we have not upgraded to rules_nodejs 5.x yet (using 4.6).

What I'm hoping to do (something similar to):

genrule(
    name = "playwright_test_runner",
    srcs = [
        ":playwright_browsers",
    ]),
    outs = [
        "playwright_runner.sh",
    ],
    cmd = """
echo -ne '
#!/bin/sh
set -euo pipefail

# Untar browsers
tar xf $(location :playwright_browsers)

PLAYWRIGHT_BROWSERS_PATH=browsers $(execpath :playwright) test -c $$1  --reporter=dot
' > $@ """,
    tools = [
        "@playwright_npm//@playwright",
    ],
)

:playwright_browsers is a genrule that outputs a browsers.tar file which is the browsers playwright needs to run the browser tests

And then I'd want to call the outputted shell script with a sh_test with arguments that will set the playwright config file and ensure we have our test data in a data = glob("**/*.ts"). It doesn't seem like I can use the generated playwright index.bzl files directly because we need to download the browsers and tar them up (since you need to declare every file output which would be too hard).

This seems close enough to this ticket so I'm asking here, but please lmk if there is a better place for this!

[Zemnmez]

run_node seems good but it appears to be an action, not a macro and I find actions kind of complicated :'(

[alexeagle]

rules_js is nearing 1.0 and solves this problem. js_binary can be used in a genrule https://github.com/aspect-build/rules_js/blob/main/examples/genrule/BUILD.bazel there is no more run_node helper for actions either, you just use standard ctx.actions.run.

@Zemnmez there's an example for macros too https://github.com/aspect-build/rules_js/blob/main/examples/macro/mocha.bzl

I don't think anyone is going to fix this in rules_nodejs as it's very breaking.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 6 months. It will be closed if no further activity occurs in 30 days. Collaborators can add a "cleanup" or "need: discussion" label to keep it open indefinitely. Thanks for your contributions to rules_nodejs!

[github-actions[bot]]

This issue was automatically closed because it went 30 days without any activity since it was labeled "Can Close?"