Missing multistage dependencies

[pcoves]

Actual behavior

The built image using kaniko is missing dependencies while it works just fine with docker

Expected behavior

The built image must ship my software dependencies

To Reproduce

1. clone https://gitlab.com/pcoves/pcoves.gitlab.io && cd pcoves.gitlab.io
2. docker run --rm -v $(pwd):/workspace -w /workspace registry.gitlab.com/pcoves/pcoves.gitlab.io:latest build (This works)
3. docker run --rm -v $(pwd):/workspace -w /workspace registry.gitlab.com/pcoves/pcoves.gitlab.io:kaniko build (This fails)

Additional Information

• Dockerfile

FROM haskell:latest as build

COPY . /pcoves.gitlab.io
WORKDIR /pcoves.gitlab.io
RUN stack install

FROM gcr.io/distroless/base

COPY --from=build /root/.local/bin /root/.local/bin
COPY --from=build /lib/x86_64-linux-gnu/libz.so.1 /lib/x86_64-linux-gnu/libz.so.1
COPY --from=build /usr/lib/x86_64-linux-gnu/libgmp.so.10 /usr/lib/x86_64-linux-gnu/libgmp.so.10

ENTRYPOINT ["/root/.local/bin/pcoves-gitlab-io"]

• Build Context

master:
  stage: master
  image: docker:latest
  dependencies:
    - dev
  cache:
    paths:
      - _cache
      - .stack-work
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  services:
    - docker:dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker build -t registry.gitlab.com/pcoves/pcoves.gitlab.io .
    - docker push registry.gitlab.com/pcoves/pcoves.gitlab.io
  only:
    - master

kaniko:
  stage: kaniko
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:kaniko
  only:
    - kaniko

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [X]
Please check if this error is seen when you use --cache flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [X]

[clhodapp]

I believe that this is a dupe of https://github.com/GoogleContainerTools/kaniko/issues/552, which I believe is a bigger deal than folks might realize.

[pcoves]

I believe that this is a dupe of #552, which I believe is a bigger deal than folks might realize.

Not sure.

As far as I understand #552, nothing gets copied from one stage to the other.

Here, my executable is present in the final image. Only the dependencies are missing when using kaniko.

[consokodev]

I seem have the same problem, with "--from" in Dockerfile when I build with Kaniko, actually it get content from cache not from builder

COPY --from=builder /app/app-api /app
COPY ./config/ /app/config/

It run normally with Docker daemon

[WesCossick]

One thing we noticed with some of our company's images is that /lib/x86_64-linux-gnu/libz.so.1 is a symlink to /lib/x86_64-linux-gnu/libz.so.1.2.8 (your specific version may vary).

Because of that, this command...

COPY --from=build /lib/x86_64-linux-gnu/libz.so.1 /lib/x86_64-linux-gnu/libz.so.1

...works just fine with Docker, but does not seem to work with kaniko. We didn't do a whole lot of testing as to why—I assume it's either not copying anything at all, or just copying the symlink without copying the file the symlink points to.

To fix the issue, we simply changed the command to:

COPY --from=build /lib/x86_64-linux-gnu/libz.so.1.2.8 /lib/x86_64-linux-gnu/libz.so.1

Anyway, if some lib dependencies don't seem to be present after copying from a previous stage in a multi-stage build, it's could be due to how kaniko is handling symlinked files.

Something the kaniko team may want to look into, because the behavior appears to deviate from Docker.

[tejal29]

Thank you @pcoves for the issue. We would love community contributions on this one. Unfortunately, we dont have anyone actively working on this.