Node.js binary not included in built image

[micthiesen]

Actual behavior We are using kaniko to build an image within GitLab CI using GitLab runners. The image is built successfully and pushed to our registry. When attempting to use the image, running a command installed via npm, it fails with the following error: /usr/bin/env: ‘node’: No such file or directory

If we open a bash shell inside the image and look for the node or npm binaries, they are not found:

root@12fb111175f7:/# which node
root@12fb111175f7:/# npm
bash: npm: command not found

The binaries are not found in /usr/bin/ which is usually where they are installed.

Expected behavior The node and npm binaries should be installed inside the image. When using Docker on our local machine they are.

To Reproduce Steps to reproduce the behavior:

1. Use kaniko to build an image which installs Node.js (e.g. the one included below)
2. Look for the node and npm binaries inside the container. They will be missing

Additional Information

• Dockerfile Please provide either the Dockerfile you're trying to build or one that can reproduce this error:

FROM python:3.7.3

# Add an APT source for LTS Node.js. Now includes npm
# https://github.com/nodesource/distributions/blob/master/README.md#installation-instructions
RUN curl --silent --location https://deb.nodesource.com/setup_12.x | bash

RUN apt-get update --quiet --yes \
 && apt-get install \
    --no-install-recommends \
    --yes \
    libmemcached-dev \
    nodejs
RUN npm install --global snyk

• Build Context Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands): The Dockerfile only contains a single FROM and 3 RUN commands
• Kaniko Image (fully qualified with digest): sha256:260ca7bf7866d0d59c8ce83b66af231e888f44b00c7c738fe5dc9207524cc68c for gcr.io/kaniko-project/executor:debug

[micthiesen]

Enabling caching seems to bypass this bug. This is the command we now use:

/kaniko/executor
    --cache true
    --cache-ttl 12h
    --context ${CI_PROJECT_DIR}
    --dockerfile ${CI_PROJECT_DIR}/${DOCKERFILE_NAME}
    --destination ${DESTINATION}

[tejal29]

@micthiesen Thanks for providing the work-around. We will definitely investigate and fix this.

[micthiesen]

@tejal29 thanks. I’d like to add that I’ve had inconsistent results trying to reproduce this. I’m uncertain now whether or not it is the caching that fixed things. Good luck with the investigation

[rustymunkey]

We stumbled upon a similar issue when installing the "graphviz" debian package. The kaniko build succeeds, however in the resulting image several files are missing.

Our Dockerfile:

ARG CONFLUENCE_VERSION=6.15.9
FROM atlassian/confluence-server:${CONFLUENCE_VERSION}-ubuntu

RUN apt-get update -y \
    && DEBIAN_FRONTEND="noninteractive" apt-get install -y tzdata ttf-dejavu graphviz \
    && apt-get clean

The result is not deterministic. Sometimes newly installed libraries are missing, sometimes the "/usr/bin/dot" binary is not in the image.

We appended an ls command at the end of the RUN to see if the libraries were installed, and - yes - all required libs were at the correct place. Thus, the problem must be sometimes after the RUN commands were executed.

We are running kaniko in a docker container, as we intend to use it in our CI pipeline do build docker images without the need of accessing the docker server.

We tried several workarounds, including caching, --single-snapshot or --snapshotMode time.

[donmccasland]

Can one of you add a log of a debug run of kaniko that reproduces the issue?

[rustymunkey]

At least on our side it was the problem with our kaniko docker image. We took great care in creating it (yes, we have seen the warning on the readme!), as it would better fit to our build pipeline. However, when we switched back to the official docker image, these weird non-deterministic problems disappeard and never came back. After several weeks of testing and building various images with kaniko, I think the problem can be seen as solely from our custom docker image

[tejal29]

@micthiesen and @rustymunkey I think we might have identified the root cause for this behavior. The reason why this is intermittent is,

• apt get ends up running child process in other process groups.
• kaniko isnt aware of these and does not wait for them to complete at times.

We are working on investigating this further. I will update this ticket with progress. Hopefully you will be game for switching back to kaniko when this issue is fixed.

Thanks! And sorry for the inconvenience. Tejal.

[drptbl]

@tejal29 just encountered this issue after installing yarn and building this docker container with kaniko. All files from yarn unpacked below are missing.

FROM node:10.19.0-buster
ENV YARN_VERSION 1.22.0
# install latest yarn
RUN curl -fSLO --compressed "https://yarnpkg.com/downloads/$YARN_VERSION/yarn-v$YARN_VERSION.tar.gz" \
    && tar -xzf yarn-v$YARN_VERSION.tar.gz -C /opt/ \
    && ln -snf /opt/yarn-v$YARN_VERSION/bin/yarn /usr/local/bin/yarn \
    && ln -snf /opt/yarn-v$YARN_VERSION/bin/yarnpkg /usr/local/bin/yarnpkg \
    && rm yarn-v$YARN_VERSION.tar.gz

@scone

[tejal29]

Thanks @drptbl . this is the next on our list. Will update the progress

[tejal29]

@drptbl I tried your dockerfile and am not able to reproduce this error.

tejaldesai@tejaldesai-macbookpro2 ~docker run -it  gcr.io/tejal-test/test /bin/sh
tejaldesai@tejaldesai-macbookpro2 ~docker run -it  gcr.io/tejal-test/test /bin/sh
# which npm 
/usr/local/bin/npm
# ls /usr/local/bin/yarnpkg
/usr/local/bin/yarnpkg
# ls -al opt/yarn-v1.22.0/lib/
total 5200
drwxr-xr-x 2 root root    4096 Feb  5 12:54 .
drwxr-xr-x 4 root root    4096 May  6 05:44 ..
-rwxr-xr-x 1 root root 5300709 Feb  5 12:54 cli.js
-rw-r--r-- 1 root root    9910 Feb  5 12:54 v8-compile-cache.js
# ls -al /opt/yarn-v1.22.0/bin/
total 28
drwxr-xr-x 2 root root 4096 Feb  5 12:54 .
drwxr-xr-x 4 root root 4096 May  6 05:44 ..
-rwxr-xr-x 1 root root 1025 Feb  5 12:54 yarn
-rwxr-xr-x 1 root root   34 Feb  5 12:54 yarn.cmd
-rwxr-xr-x 1 root root 1015 Feb  5 12:54 yarn.js
-rwxr-xr-x 1 root root   42 Feb  5 12:54 yarnpkg
-rwxr-xr-x 1 root root   30 Feb  5 12:54 yarnpkg.cmd
# 

But it could be due to its non-deterministic nature.

[POD666]

I also have this issue with custom kaniko image:

My case

Image with kaniko binary:

FROM docker:20.10.8 as docker

FROM gcr.io/kaniko-project/executor:v1.3.0-debug as kaniko

FROM python:3.6-slim

COPY --from=docker /usr/local/bin/docker /usr/local/bin/
COPY --from=kaniko /kaniko/executor /kaniko/

# Pre-install helpful commands that are used in CI in most cases
RUN apt-get update -y && \
    apt-get install -y -qq graphviz bzip2 git make curl moreutils netcat && \
    pip install ...

And then when I build another image:

ARG DOCKER_IMAGE_BASE=python:3.6
FROM ${DOCKER_IMAGE_BASE}

# instrumentation:
RUN pip install -U tox pytest sh ipython tox awscli 'pip>=20,<20.3'

it results in an image with missing binaries:

/scripts-7965327-1635171869/step_script: line 324: tox: command not found
/scripts-7965327-1635171869/step_script: line 324: ts: command not found

Running apt-get install moreutils inside that broken image says everything is installed and I can get binaries by running with --reinstall flag. (it's not a workaround at all)

Looks like it worked fine with another custom image:

FROM gcr.io/kaniko-project/executor:v1.3.0-debug as kaniko

FROM docker:20.10.2

COPY --from=kaniko /kaniko/executor /kaniko/

RUN apk --no-cache add py3-pip python3-dev libffi-dev openssl-dev gcc libc-dev make curl moreutils

RUN pip install -U 'pip>=20,<20.3' 'cryptography<3.4' awscli --no-cache-dir

And the issue appeared when I switched base image to python:3.6-slim.

Also, this seems to be related: https://github.com/GoogleContainerTools/kaniko/issues/1714

Is there any workaround?

[federico-manuel-trasande-kyndryl]

Hi, I've also had this issue where the image gets build but I found binaries are missing.

My dockerfile:

FROM docker.io/node:18-alpine3.20

ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD 1
ENV PUPPETEER_EXECUTABLE_PATH /usr/bin/chromium-browser
ENV CHROME_BIN /usr/bin/chromium-browser
RUN apk add --no-cache openjdk17-jdk chromium-chromedriver tini make gcc g++ git python3 && rm -rf /var/cache/apk/* && \
[ -L /usr/lib/libjli.so ] || ln -s /usr/lib/jvm/java-17-openjdk/lib/libjli.so /usr/lib/libjli.so
USER node

I'm using gcr.io/kaniko-project/executor:debug to build the image. I've found that for some reason java binary is missing but if check apk package I see it as installed, if I build the image locally with docker it doesn't have the missing bin.