multistage build enabling and using busybox to enable RUN fails in multi stage build works with docker

[MikeMoore63]

Actual behavior When building a multistage image and using busybox as a shell in secondary stage works when using docker but fails when using kaniko.

Preparing to unpack .../busybox_1%3a1.30.1-4ubuntu6.4_amd64.deb ...
Unpacking busybox (1:1.30.1-4ubuntu6.4) ...
Setting up busybox (1:1.30.1-4ubuntu6.4) ...
INFO[0014] Taking snapshot of full filesystem...        
INFO[0015] Saving file usr/lib for later use            
INFO[0016] Saving file lib for later use                
INFO[0016] Saving file usr/lib64 for later use          
INFO[0016] Saving file lib64 for later use              
INFO[0016] Saving file etc for later use                
INFO[0016] Saving file usr/lib/os-release for later use 
INFO[0016] Saving file usr/lib/ssl for later use        
INFO[0016] Saving file usr/lib/locale for later use     
INFO[0016] Saving file home for later use               
INFO[0016] Saving file usr/share/ca-certificates for later use 
INFO[0016] Saving file bin/busybox for later use        
INFO[0016] Deleting filesystem...                       
INFO[0016] No base image, nothing to extract            
INFO[0016] Executing 0 build triggers                   
INFO[0016] Building stage 'scratch' [idx: '1', base-idx: '-1'] 
INFO[0016] Unpacking rootfs as cmd ADD tmp.tgz / requires it. 
INFO[0016] Using files from context: [/workspace/tmp.tgz] 
INFO[0016] ADD tmp.tgz /                                
INFO[0016] Unpacking local tar archive tmp.tgz to /     
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /lib /lib/                 
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /lib64 /lib64/             
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /etc /etc/                 
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /usr/lib/os-release /usr/lib/os-release 
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /usr/lib/ssl /usr/lib/ssl/ 
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /usr/lib/locale /usr/lib/locale/ 
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /home /home/               
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /usr/share/ca-certificates /usr/share/ca-certificates 
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /bin/busybox /bin/busybox  
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /bin/busybox /bin/sh       
INFO[0016] Taking snapshot of files...                  
INFO[0016] RUN  /bin/busybox ln -s /bin/busybox /bin/ln; ln -s /bin/busybox /bin/rm  ; for i in $(/bin/busybox --list); do if [ -f /bin/$i ]; then /bin/busybox rm /bin/$i; fi; if [ ! -L /bin/$i ]; then /bin/busybox ln -s /bin/busybox /bin/$i;fi;done;mkdir -p /usr/bin;/bin/busybox ln -s /bin/busybox /usr/bin/env 
INFO[0016] Initializing snapshotter ...                 
INFO[0016] Taking snapshot of full filesystem...        
INFO[0016] Cmd: /bin/sh                                 
INFO[0016] Args: [-c /bin/busybox ln -s /bin/busybox /bin/ln; ln -s /bin/busybox /bin/rm  ; for i in $(/bin/busybox --list); do if [ -f /bin/$i ]; then /bin/busybox rm /bin/$i; fi; if [ ! -L /bin/$i ]; then /bin/busybox ln -s /bin/busybox /bin/$i;fi;done;mkdir -p /usr/bin;/bin/busybox ln -s /bin/busybox /usr/bin/env] 
INFO[0016] Running: [/bin/sh -c /bin/busybox ln -s /bin/busybox /bin/ln; ln -s /bin/busybox /bin/rm  ; for i in $(/bin/busybox --list); do if [ -f /bin/$i ]; then /bin/busybox rm /bin/$i; fi; if [ ! -L /bin/$i ]; then /bin/busybox ln -s /bin/busybox /bin/$i;fi;done;mkdir -p /usr/bin;/bin/busybox ln -s /bin/busybox /usr/bin/env] 
error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: no such file or directory

Expected behavior Build to work like it does with Docker

To Reproduce Steps to reproduce the behavior:

1. Create docker file
2. create bash script kaniko_link_fixer.sh to avoid issue with symlinks and kaniko
3. Build with "docker run -it --rm -v ${PWD}:/workspace -v "$HOME/.config/gcloud:/root/.config/gcloud" gcr.io/kaniko-project/executor:latest --no-push"

Additional Information

• Dockerfile

FROM gcr.io/cloud-marketplace/google/ubuntu2004 as build
COPY kaniko_link_fixer.sh .
RUN  apt-get update && \
   apt-get install -y \
     --no-install-recommends busybox && \
   ./kaniko_link_fixer.sh

FROM scratch
ADD tmp.tgz /
# create a basic set up so linux will work
COPY --from=build /lib /lib/
COPY --from=build /lib64 /lib64/
# some basic data files
COPY --from=build /etc /etc/
COPY --from=build /usr/lib/os-release /usr/lib/os-release
COPY --from=build /usr/lib/ssl /usr/lib/ssl/
COPY --from=build /usr/lib/locale /usr/lib/locale/
# COPY --from=build /usr/local/bin/grpc /usr/local/bin/grpc/
COPY --from=build /home /home/
COPY --from=build /usr/share/ca-certificates /usr/share/ca-certificates

# debug can be removed in non debug
COPY --from=build /bin/busybox /bin/busybox
COPY --from=build /bin/busybox /bin/sh

RUN  /bin/busybox ln -s /bin/busybox /bin/ln; ln -s /bin/busybox /bin/rm  ; for i in $(/bin/busybox --list); do if [ -f /bin/$i ]; then /bin/busybox rm /bin/$i; fi; if [ ! -L /bin/$i ]; then /bin/busybox ln -s /bin/busybox /bin/$i;fi;done;mkdir -p /usr/bin;/bin/busybox ln -s /bin/busybox /usr/bin/env

• Build Context kaniko_link_fixer.sh used to remove symlinks and replace with files so copy down works

#!/usr/bin/env bash
# so kaniko breaks when you copy to second layers in multi-stage build
# https://github.com/GoogleContainerTools/kaniko/issues/2381
# this is hopefully a work around
# basically this resolves symlinks for specific directories that block multi stage
# find links that point at directories
# replace the directory links with directory 
# copy cntents recursively into new directory
# we do this stage first to get to just file links
# only do this if running in kaniko detected directory kaniko exists
if [ -d /kaniko ]
then
for f in $(find /usr/lib/ssl /etc/ssl  -type l )
do
  export a=$(/usr/bin/readlink -f $f)
  if [  -d "$f" ]
  then
      rm "$f"
      mkdir -p "$f"
      cp -r "$a" "$f"
  fi
done

# now we should just have file links we realise the links to the actual files
# this does duplicate data so costs disk space
for f in $(find /usr/lib/ssl /etc/ssl  -type l )
do
  export a=$(/usr/bin/readlink -f $f)
  if [ -f "$f" ]
  then
      rm "$f"
      cp "$a" "$f"
  else
      echo "should not happen!!!!!!!!!! $f is not a file"
  fi
done
fi

You will also need a tar tmp.tgz contents not that important in my case I have directory /tmp with 1777 permissions to provide a /tmp

• Kaniko Image (fully qualified with digest) gcr.io/kaniko-project/executor:latest sha256:23ae6ccaba2b0f599966dbc5ecf38aa4404f4cd799add224167eaf285696551a

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [ ]
Please check if this error is seen when you use --cache flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [ ]

[Ayushmaanseth]

Hi, is this still being looked into or has this been solved? Facing a similar kind of issue myself