Kaniko fails to unpack an image if symlink to a directory was replaced with a new dir and its old target removed

[uvlad7]

Actual behavior A clear and concise description of what the bug is.

I get an error

INFO[0000] Building stage 'localhost:5000/intermediate:4' [idx: '0', base-idx: '-1'] 
INFO[0000] Unpacking rootfs as cmd RUN touch grass requires it. 
error building image: error building stage: failed to get filesystem from image: mkdir /things: file exists

Maybe it's the same as #3428

Expected behavior A clear and concise description of what you expected to happen.

Dir should be created successfully

To Reproduce Steps to reproduce the behavior:

1. Create a dir with the following structure

$ tree                                                                                                                                                                                                                                                   
.
├── base
│   └── Dockerfile
├── boom
│   └── Dockerfile
├── build.sh
└── intermediate
    └── Dockerfile

3 directories, 4 files
$ cat build.sh                                                                                                                                                                                                                                           
#!/usr/bin/env bash
set -e
# docker run -d -p 5000:5000 --name registry registry:2
dirs=(base intermediate boom)

version=$(getfattr -n user.build_ver --only-values $0 2>/dev/null || echo '0')
version=$((version + 1))
setfattr -n user.build_ver -v $version $0
echo "build version: $version"

for dir in ${dirs[@]}; do
    cd $dir
    docker run --network=host --rm -v $PWD:/workspace gcr.io/kaniko-project/executor@sha256:9e69fd4330ec887829c780f5126dd80edc663df6def362cd22e79bcdf00ac53f --context . --cache-run-layers=false --dockerfile "/workspace/Dockerfile" "--destination=localhost:5000/$dir:$version" --build-arg "VERSION=$version"
    cd ..
done
$ cat base/Dockerfile                                                                                                                                                                                                                                    
# Just because I have it locally
FROM buildpack-deps:bookworm
RUN mkdir stuff && echo "42" > stuff/answer.h && ln -s stuff things
$ cat intermediate/Dockerfile 
ARG VERSION
FROM localhost:5000/base:${VERSION}
RUN rm things && rm -r stuff && mkdir things && echo "34" > things/answer.c
vladimir@np940x5n:~/profit/kanibug$ cat boom/Dockerfile                                                                                                                                                                                                                                    
ARG VERSION
FROM localhost:5000/intermediate:${VERSION}
RUN touch grass

Note:

this also fails

ARG VERSION
FROM localhost:5000/base:${VERSION}
RUN rm things && mkdir things
RUN rm -r stuff && echo "34" > things/answer.c

but remove in different steps

ARG VERSION
FROM localhost:5000/base:${VERSION}
RUN rm things
RUN mkdir things
RUN rm -r stuff && echo "34" > things/answer.c

or

ARG VERSION
FROM localhost:5000/base:${VERSION}
RUN rm things && rm -r stuff
RUN mkdir things && echo "34" > things/answer.c

works, also it works if target dir is kept

ARG VERSION
FROM localhost:5000/base:${VERSION}
RUN rm things && mkdir things && echo "34" > things/answer.c

2. run ./build.sh

Additional Information

• Dockerfile Please provide either the Dockerfile you're trying to build or one that can reproduce this error. Multiple files above
• Build Context Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands) No ADD/COPY commands used
• Kaniko Image (fully qualified with digest) gcr.io/kaniko-project/executor@sha256:9e69fd4330ec887829c780f5126dd80edc663df6def362cd22e79bcdf00ac53f

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [ ]
Please check if this error is seen when you use --cache flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [ ]

[uvlad7]

A real-life example is

FROM buildpack-deps:bookworm

RUN apt-get update -qq -y --allow-unauthenticated && \
    wget https://repo.percona.com/apt/percona-release_latest.bookworm_all.deb && \
    apt install -y ./percona-release_latest.bookworm_all.deb && \
    rm percona-release_latest.bookworm_all.deb && \
    percona-release setup ps57 && apt install -y libperconaserverclient20-dev

buildpack-deps:bookworm has /usr/include/mysql symlink to mariadb, apt install -y libperconaserverclient20-dev both removes /usr/include/mariadb and creates dir /usr/include/mysql; so this image cannot be used in kaniko, no matter if it's built in kaniko itself of docker

[mzihlmann]

i experienced the same issue originally when installing percona, i dumbed it down to the manually create link, should have left you a breadcrumb in the issue description.

[mzihlmann]

If you're still looking for a solution to this you could give my fork a try https://github.com/mzihlmann/kaniko/releases/ It fixes this issue and a few more, mostly related to caching, if you have other issues you would like to see resolved please let me know. I know that this is not ideal and I hope we can get the changes merged here eventually but for now that's the best I can offer. If you like what you see you can support me with a star, thank you 🙇