failed tool installation but still considered installed

[korosuke613]

Summary

When installing a new version of Node.js, the file download fails for some reason. However, an empty directory will remain to store the files for that version. Whether or not a version of Node.js is already installed is determined by the presence or absence of that version's directory. Thus, even if Node.js is not installed, it is assumed to be installed, and subsequent processing will fail due to missing files.

Details

We are self-hosting Renovate, which depends on ghcr.io/containerbase/buildpack image.

Recently that Renovate has been continually encountering the following error and has been unable to update dependencies that depend on Node.js v18.14.0. The same issue occurred when Node.js v18.13.0 was released.

DEBUG: Resolved stable matching version (repository=dummy/dummy-repo, baseBranch=develop, branch=renovate/dummy/dummy-2.x)
       "toolName": "npm",
       "constraint": "<9",
       "resolvedVersion": "8.19.3"
DEBUG: Executing command (repository=dummy/dummy-repo, baseBranch=develop, branch=renovate/dummy/dummy-2.x)
       "command": "install-tool node v18.14.0"
DEBUG: rawExec err (repository=dummy/dummy-repo, baseBranch=develop, branch=renovate/dummy/dummy-2.x)
       "err": {
         "name": "ExecError",
         "cmd": "/bin/sh -c install-tool node v18.14.0",
         "stderr": "No /opt/buildpack/tools/node/18.14.0/bin defined - aborting\n",
         "stdout": "tool node v18.14.0 is already installed\nlinking tool node v18.14.0\n",
         "options": {
           "cwd": "/tmp/renovate/github/dummy/dummy-repo/hoge/fuga/applications/system-admin",
           "encoding": "utf-8",
           "env": {
             "NPM_CONFIG_CACHE": "/tmp/renovate/cache/others/npm",
             "npm_config_store": "/tmp/renovate/cache/others/pnpm",
             "HOME": "/home/ubuntu",
             "PATH": "/home/ubuntu/.cargo/bin:/home/ubuntu/.local/bin:/go/bin:/opt/buildpack/tools/python/3.11.1/bin:/home/ubuntu/.npm-global/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
             "LC_ALL": "C.UTF-8",
             "LANG": "C.UTF-8",
             "BUILDPACK_CACHE_DIR": "/tmp/renovate/containerbase",
             "CONTAINERBASE_CACHE_DIR": "/tmp/renovate/containerbase"
           },
           "maxBuffer": 10485760,
           "timeout": 900000
         },
         "exitCode": 1,
         "message": "Command failed: install-tool node v18.14.0\nNo /opt/buildpack/tools/node/18.14.0/bin defined - aborting\n",
         "stack": "ExecError: Command failed: install-tool node v18.14.0\nNo /opt/buildpack/tools/node/18.14.0/bin defined - aborting\n\n    at ChildProcess.<anonymous> (/usr/src/app/node_modules/renovate/dist/util/exec/common.js:87:24)\n    at ChildProcess.emit (node:events:525:35)\n    at ChildProcess.emit (node:domain:489:12)\n    at ChildProcess._handle.onexit (node:internal/child_process:291:12)"
       }

stdout says tool node v18.14.0 is already installed linking tool node v18.14.0, but stderr says No /opt/buildpack/tools/node/18.14.0/bin defined - aborting.

I checked the existence of the files in the docker container and found that the directory /opt/buildpack/tools/node/18.14.0/ existed, but the directory was empty.

Further back in the logs, I found that /bin/sh -c install-tool node 18.14.0 failed with a 404 error.

DEBUG: Executing command (repository=dummy/dummy-repo, branch=renovate/major-node)
       "command": "install-tool node 18.14.0"
DEBUG: rawExec err (repository=dummy/dummy-repo, branch=renovate/major-node)
       "err": {
         "name": "ExecError",
         "cmd": "/bin/sh -c install-tool node 18.14.0",
         "stderr": "curl: (22) The requested URL returned error: 404 \nDownload failed: https://nodejs.org/dist/v18.14.0/node-v18.14.0-linux-x64.tar.xz\n",
         "stdout": "installing v2 tool node v18.14.0\n",
         "options": {
           "cwd": "/tmp/renovate/github/dummy/dummy-repo",
           "encoding": "utf-8",
           "env": {
             "NPM_CONFIG_CACHE": "/tmp/renovate/cache/others/npm",
             "npm_config_store": "/tmp/renovate/cache/others/pnpm",
             "HOME": "/home/ubuntu",
             "PATH": "/home/ubuntu/.cargo/bin:/home/ubuntu/.local/bin:/go/bin:/opt/buildpack/tools/python/3.11.1/bin:/home/ubuntu/.npm-global/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
             "LC_ALL": "C.UTF-8",
             "LANG": "C.UTF-8",
             "BUILDPACK_CACHE_DIR": "/tmp/renovate/containerbase",
             "CONTAINERBASE_CACHE_DIR": "/tmp/renovate/containerbase"
           },
           "maxBuffer": 10485760,
           "timeout": 900000
         },
         "exitCode": 1,
         "message": "Command failed: install-tool node 18.14.0\ncurl: (22) The requested URL returned error: 404 \nDownload failed: https://nodejs.org/dist/v18.14.0/node-v18.14.0-linux-x64.tar.xz\n",
         "stack": "ExecError: Command failed: install-tool node 18.14.0\ncurl: (22) The requested URL returned error: 404 \nDownload failed: https://nodejs.org/dist/v18.14.0/node-v18.14.0-linux-x64.tar.xz\n\n    at ChildProcess.<anonymous> (/usr/src/app/node_modules/renovate/dist/util/exec/common.js:87:24)\n    at ChildProcess.emit (node:events:525:35)\n    at ChildProcess.emit (node:domain:489:12)\n    at ChildProcess._handle.onexit (node:internal/child_process:291:12)"
       }

I got to containerbase/base. install_tool in tools/v2/node.sh leaves an empty directory when it fails to download files after creating a new directory. https://github.com/containerbase/base/blob/cef6d83959eba00a2d6df79000824393d16f7ab7/src/usr/local/buildpack/tools/v2/node.sh#L39-L51

And if there is an empty directory, check_tool_installed will pass, and even an empty directory will be considered installed. https://github.com/containerbase/base/blob/cef6d83959eba00a2d6df79000824393d16f7ab7/src/usr/local/buildpack/utils/v2/defaults.sh#L19-L22

Workaround

I deleted the empty directory in the container and it started downloading again, successfully making the new version of Node.js available.

[viceice]

this can also happen on other tools, so i updated the title.

[heyLu]

Looks like https://github.com/renovatebot/renovate/discussions/29999#discussioncomment-10023299 is also a case of this.

In our renovate-bot install it seems the Golang download got a 404 response, which resulted in an empty /tmp/renovate/cache/containerbase/<hash>/go1.22.5.linux-amd64.tar.gz file being written. That made the Golang install fail every time, as there was no attempt to redownload the .tar.gz file.

[viceice]

will start fixing on Monday

[viceice]

should be fixed in v10.16.0