NodeJS images don't work in readonly filesystems

[codersasha]

🐞 bug report

Affected Rule

The issue is caused by the rule: nodejs_image and other nodejs rules.

Is this a regression?

No.

Description

nodejs_image requires a writeable filesystem. Most container environments will not run containers with writeable filesystems. The kubernetes settings readOnlyRootFilesystem: true and runAsNonRoot: true are standard requirements for production-ready workloads.

Running without a writeable filesystem gives the error:

[link_node_modules.js] An error has been reported: [Error: EROFS: read-only file system, mkdir 'node_modules'] {
  errno: -30,
  code: 'EROFS',
  syscall: 'mkdir',
  path: 'node_modules'
} Error: EROFS: read-only file system, mkdir 'node_modules'

as nodejs runs the linker and tries to create a node_modules folder with symlinks: https://github.com/bazelbuild/rules_nodejs/blob/stable/internal/node/launcher.sh#L231

Without this, Bazel nodejs images are not secure enough to run in production systems.

🔬 Minimal Reproduction

Create a nodejs_image rule, e.g.:

nodejs_image(
    name = "image",
    binary = ":app",
    # These two parameters are needed until bazelbuild/rules_docker#2078 is fixed.
    include_node_repo_args = False,
    node_repository_name = "nodejs_linux_amd64",
)

then run:

docker run --read-only bazel/path/to/nodejs/image

And you should see the error.

🔥 Exception or Error

[link_node_modules.js] An error has been reported: [Error: EROFS: read-only file system, mkdir 'node_modules'] {
  errno: -30,
  code: 'EROFS',
  syscall: 'mkdir',
  path: 'node_modules'
} Error: EROFS: read-only file system, mkdir 'node_modules'

Disabling the linker does fix the issue:

nodejs_image(
    name = "image",
    binary = ":app",
    # These two parameters are needed until bazelbuild/rules_docker#2078 is fixed.
    include_node_repo_args = False,
    node_repository_name = "nodejs_linux_amd64",
    templated_args = [
        "--nobazel_run_linker",
        "--nobazel_node_patches",
    ],
)

But this breaks imports of local dependencies, since the node linker is what creates the symlinks in the locations needed for require() to discover them. So produces errors like:

Error: Cannot find module '@localdep/config'. Please verify that the package.json has a valid "main" entry
    at Function.module.constructor._resolveFilename (path/to/nodejs/image.binary_require_patch.cjs:704:17)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (services/url-shortener/server/logger/index.js:2:16)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)

🌍 Your Environment

Operating System:

  
Linux x64
  

Output of bazel version:

  
5.2.0
  

Rules_docker version:

  
0.25.0
  

Anything else relevant?

[codersasha]

I came up with a fix for this, but it involves loading the NodeJS binary into the container twice - once with a loader script that forces an immediate exit using node --version (but runs the linker prework), and one without the linker that uses the node_modules folder created by that step.

load("@build_bazel_rules_nodejs//:index.bzl", "js_library")
load("@io_bazel_rules_docker//nodejs:image.bzl", "nodejs_image")
load("@io_bazel_rules_docker//docker/util:run.bzl", "container_run_and_commit")

js_library(
    name = "my_nodejs_lib",
    ...
)

nodejs_image(
    name = "my_nodejs_img_base",
    data = [
        ":my_nodejs_lib",
    ],
    entry_point = ":path/to/index.js",
    # These two parameters are needed until bazelbuild/rules_docker#2078 is fixed.
    include_node_repo_args = False,
    node_repository_name = "nodejs_linux_amd64",
    templated_args = [
        # Exit immediately.
        # We need this so that the script runs the linker but then exits.
        "--node_options=--version",
    ],
)

container_run_and_commit(
    name = "my_nodejs_img_with_linked_node_modules",
    commands = [
        # This does nothing except run our immediately-exiting script that links the node_modules.
        "",
    ],
    image = ":my_nodejs_img_base.tar",
)

nodejs_image(
    name = "my_nodejs_img",
    base = ":my_nodejs_img_with_linked_node_modules",
    data = [
        ":my_nodejs_lib",
    ],
    entry_point = ":path/to/index.js",
    env = {
        # Here we reference the location of the node_modules symlinked in the base.
        "NODE_PATH": "../../my_nodejs_img_base.binary.runfiles/__main__/node_modules",
    },
    # These two parameters are needed until bazelbuild/rules_docker#2078 is fixed.
    include_node_repo_args = False,
    node_repository_name = "nodejs_linux_amd64",
    templated_args = [
        # These are needed to prevent the error 'EROFS: read-only file system, mkdir 'node_modules''.
        # We don't run the linker this time, because it ran before.
        "--nobazel_run_linker",
        # Need this so Bazel doesn't write over the node binary, which would also fail on a read-only FS.
        "--nobazel_node_patches",
        # Need this so Bazel uses our NODE_PATH env variable and not its own one.
        "--nobazel_patch_module_resolver",
    ],
)

It's not ideal, but the idea is sound - run the launcher script in a container_run_and_commit (or, ideally, a layer) then put the final image (with a "read-only filesystem"-safe launcher script) on top of it.

[codersasha]

Following up - found a better way to do it without two nodejs_image rules, which means node only ends up being in the final image once, but it involves patching rules_nodejs to have a new environment variable that allows us to re-use the launcher script at both steps.

WORKSPACE:

http_archive(
    name = "build_bazel_rules_nodejs",
    patch_args = ["-p1"],
    patches = [
        # Add support for BAZEL_NODE_RUN_LINKER_ONLY, a flag in the launcher script that allows
        # the node app to be prepared for execution in read-only mode.
        "//:patches/build_bazel_rules_nodejs_read_only_launcher_script.patch",
    ],
    sha256 = "958554fcf0a9200327569322cfb56f50c6d1ba430b80312687fe37c19429f9b8",
    strip_prefix = "rules_nodejs-5.5.1",
    urls = ["https://github.com/bazelbuild/rules_nodejs/archive/refs/tags/5.5.1.tar.gz"],
)

patches/build_bazel_rules_nodejs_read_only_launcher_script.patch:

diff --git a/internal/node/launcher.sh b/internal/node/launcher.sh
index 9d1e8df4..14e96e8d 100644
--- a/internal/node/launcher.sh
+++ b/internal/node/launcher.sh
@@ -182,6 +182,7 @@ STDOUT_CAPTURE=""
 STDERR_CAPTURE=""
 EXIT_CODE_CAPTURE=""
 NODE_WORKING_DIR=""
+BAZEL_NODE_RUN_LINKER_ONLY="${BAZEL_NODE_RUN_LINKER_ONLY:-}"
 
 RUN_LINKER=true
 NODE_PATCHES=true
@@ -222,7 +223,7 @@ for ARG in ${ALL_ARGS[@]+"${ALL_ARGS[@]}"}; do
 done
 
 # Link the first-party modules into node_modules directory before running the actual program
-if [ "$RUN_LINKER" = true ]; then
+if [[ "$RUN_LINKER" = true || -n "$BAZEL_NODE_RUN_LINKER_ONLY" ]]; then
   link_modules_script=$(rlocation "TEMPLATED_link_modules_script")
   # For programs which are called with bazel run or bazel test, there will be no additional runtime
   # dependencies to link, so we use the default modules_manifest which has only the static dependencies
@@ -231,7 +232,7 @@ if [ "$RUN_LINKER" = true ]; then
   "${node}" "${link_modules_script}" "${MODULES_MANIFEST:-}"
 fi
 
-if [ "$NODE_PATCHES" = true ]; then
+if [[ "$NODE_PATCHES" = true ]]; then
   node_patches_script=$(rlocation "TEMPLATED_node_patches_script")
   # Node's --require option assumes that a non-absolute path not starting with `.` is
   # a module, so that you can do --require=source-map-support/register
@@ -409,7 +410,9 @@ if [[ -n "$NODE_WORKING_DIR" ]]; then
   cd "$NODE_WORKING_DIR"
 fi
 set +e
-if [[ -n "${STDOUT_CAPTURE}" ]] && [[ -n "${STDERR_CAPTURE}" ]]; then
+if [[ -n "$BAZEL_NODE_RUN_LINKER_ONLY" ]]; then
+  "echo" <&0 &
+elif [[ -n "${STDOUT_CAPTURE}" ]] && [[ -n "${STDERR_CAPTURE}" ]]; then
   "${node}" ${LAUNCHER_NODE_OPTIONS[@]+"${LAUNCHER_NODE_OPTIONS[@]}"} ${USER_NODE_OPTIONS[@]+"${USER_NODE_OPTIONS[@]}"} "${MAIN}" ${ARGS[@]+"${ARGS[@]}"} <&0 >$STDOUT_CAPTURE 2>$STDERR_CAPTURE &
 elif [[ -n "${STDOUT_CAPTURE}" ]]; then
   "${node}" ${LAUNCHER_NODE_OPTIONS[@]+"${LAUNCHER_NODE_OPTIONS[@]}"} ${USER_NODE_OPTIONS[@]+"${USER_NODE_OPTIONS[@]}"} "${MAIN}" ${ARGS[@]+"${ARGS[@]}"} <&0 >$STDOUT_CAPTURE &
@@ -446,7 +449,7 @@ if [ "${EXPECTED_EXIT_CODE}" != "0" ]; then
       readonly BAZEL_EXIT_TESTS_FAILED=3;
       exit ${BAZEL_EXIT_TESTS_FAILED}
     fi
-  else 
+  else
     exit 0
   fi
 fi

BUILD.bazel:

load("@build_bazel_rules_nodejs//:index.bzl", "js_library")
load("@io_bazel_rules_docker//nodejs:image.bzl", "nodejs_image")
load("@io_bazel_rules_docker//container:container.bzl", "container_image")
load("@io_bazel_rules_docker//docker/util:run.bzl", "container_run_and_commit")

js_library(
    name = "my_nodejs_lib",
    ...
)

nodejs_image(
    name = "my_nodejs_img_base",
    data = [
        ":my_nodejs_lib",
    ],
    entry_point = ":path/to/index.js",
    env = {
        # Run the linker part of the launcher script, then exit immediately.
        "BAZEL_NODE_RUN_LINKER_ONLY": "1",
    },
    # These two parameters are needed until bazelbuild/rules_docker#2078 is fixed.
    include_node_repo_args = False,
    node_repository_name = "nodejs_linux_amd64",
    # Skip write operations to prevent the error "EROFS: read-only file system, mkdir 'node_modules'".
    # These are mostly ignored when BAZEL_NODE_RUN_LINKER_ONLY is set.
    templated_args = [
        # We need to skip the linker because it will try to create a node_modules directory, which would
        # make this image incompatible on a read-only filesystem.
        # Usually this would fail to resolve includes, but we'll have run the linker in a previous step
        # once the binary actually runs.
        # For more info, see bazelbuild/rules_docker#2117.
        "--nobazel_run_linker",
        # We need this so Bazel doesn't write over the node binary, which would also make this image
        # incompatible on a read-only filesystem.
        # For more info, see bazelbuild/rules_docker#1591.
        "--nobazel_node_patches",
        # We need this so Bazel uses the node_modules resolver as though the linker ran (which it did, in
        # a previous step).
        "--nobazel_patch_module_resolver",
    ],
)

container_run_and_commit(
    name = "my_nodejs_img_with_linked_node_modules",
    commands = [
        # Because the entrypoint is the node launcher script and BAZEL_NODE_RUN_LINKER_ONLY is set, this
        # links node_modules directory and exits.
        "",
    ],
    image = ":my_nodejs_img_base.tar",
)

container_image(
    name = "my_nodejs_img",
    base = ":my_nodejs_img_with_linked_node_modules",
    env = {
        # We've already linked node_modules by this point - actually run the binary this time.
        "BAZEL_NODE_RUN_LINKER_ONLY": "",
    },
)

You can test it with docker run --read-only path/to/my_nodejs:my_nodejs_img.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_docker!

[github-actions[bot]]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"