Kaniko uploads an empty cache layer

[NiklasRosenstein]

Actual behavior

I'm experiencing an issue where a cache layer retrieved from the cache repository apparently has size 0. I build variants of the image with different base images, one of them being nvcr.io/nvidia/tensorrt:22.01-py3 and the other ubuntu:bionic. I only experience this issue on the ubuntu:bionic variant.

The job where no layer is cached finishes successfully and pushes the an Artifactory Docker repository under the :_cache tag:

Successful build logs until layer that later fails

[0KUsing docker image sha256:e1e8f684a8d2023df1c21f06f2a66f7ec7ef965e6d092c7efe00ca6d592b2e6b for gcr.io/kaniko-project/executor:debug with digest gcr.io/kaniko-project/executor@sha256:3bc3f3a05f803cac29164ce12617a7be64931748c944f6c419565f500b65e8db ...[0;m
[32;1m$ set -euo pipefail; docker/development/kaniko_build.sh --destination ${DOCKER_REGISTRY}/${CANONICAL_PROJECT_PATH}/${BUILD_ARCH}-${BUILD_VARIANT}-${BUILD_TARGET}:${DOCKER_IMAGE_VERSION}[0;m
== Starting Kaniko build
BUILD_VARIANT=cpu
CANONICAL_PROJECT_PATH=my-project
ARTIFACTORY_EMAIL=[MASKED]
ARTIFACTORY_TOKEN=[MASKED]
DOCKER_REGISTRY=[MASKED]
DOCKER_USER=[MASKED]
DOCKER_PASSWORD=[MASKED]

== Base image is "ubuntu:bionic"

== Kaniko arguments:     --cache=true
    --cache-repo [MASKED]/my-project/_cache
    --dockerfile docker/development/Dockerfile
    --snapshotMode=redo
    --context /builds/my-project
    --cache-copy-layers
    --build-arg BASE_IMAGE=ubuntu:bionic
    --destination [MASKED]/my-project/amd64-cpu-application:latest

[36mINFO[0m[0000] Resolved base name ubuntu:bionic to base     
[36mINFO[0m[0000] Resolved base name base to slim              
[36mINFO[0m[0000] Resolved base name slim to development       
[36mINFO[0m[0000] Resolved base name base to application       
[36mINFO[0m[0000] Retrieving image manifest ubuntu:bionic      
[36mINFO[0m[0000] Retrieving image ubuntu:bionic from registry index.docker.io 
[36mINFO[0m[0001] Retrieving image manifest ubuntu:bionic      
[36mINFO[0m[0001] Returning cached image manifest              
[36mINFO[0m[0001] Built cross stage deps: map[1:[/opt/app]]    
[36mINFO[0m[0001] Retrieving image manifest ubuntu:bionic      
[36mINFO[0m[0001] Returning cached image manifest              
[36mINFO[0m[0001] Retrieving image manifest ubuntu:bionic      
[36mINFO[0m[0001] Returning cached image manifest              
[36mINFO[0m[0001] Executing 0 build triggers                   
[36mINFO[0m[0001] Checking for cached layer [MASKED]/my-project/_cache:db5dfc4211abf7e17354bfd97ecf6b1848fdd7070724f14a231abcef637e2266... 
[36mINFO[0m[0001] No cached layer found for cmd COPY ./docker/development/install_application_dependencies.sh /tmp/ 
[36mINFO[0m[0001] Unpacking rootfs as cmd COPY ./docker/development/install_application_dependencies.sh /tmp/ requires it. 
[36mINFO[0m[0002] ENV PYTHON_VERSION=3.6                       
[36mINFO[0m[0002] No files changed in this command, skipping snapshotting. 
[36mINFO[0m[0002] ENV LLVM_CONFIG=/usr/bin/llvm-config-10      
[36mINFO[0m[0002] No files changed in this command, skipping snapshotting. 
[36mINFO[0m[0002] ENV TZ=Europe/Berlin                         
[36mINFO[0m[0002] No files changed in this command, skipping snapshotting. 
[36mINFO[0m[0002] COPY ./docker/development/install_application_dependencies.sh /tmp/ 
[36mINFO[0m[0002] Taking snapshot of files...                  
[36mINFO[0m[0002] RUN /tmp/install_application_dependencies.sh 
[36mINFO[0m[0002] Taking snapshot of full filesystem...        
[36mINFO[0m[0002] Pushing layer [MASKED]/my-project/_cache:db5dfc4211abf7e17354bfd97ecf6b1848fdd7070724f14a231abcef637e2266 to cache now 
[36mINFO[0m[0002] Pushing image to [MASKED]/my-project/_cache:db5dfc4211abf7e17354bfd97ecf6b1848fdd7070724f14a231abcef637e2266 
[36mINFO[0m[0003] Pushed [MASKED]/my-project/_cache@sha256:6933abcf618fc9b61f6e716286e3c4197e26cff05618269af6bb8c4b0aed3906 
[36mINFO[0m[0004] cmd: /bin/sh                                 
[36mINFO[0m[0004] args: [-c /tmp/install_application_dependencies.sh] 
[36mINFO[0m[0004] Running: [/bin/sh -c /tmp/install_application_dependencies.sh] 
[ ... build output here ... ]
[36mINFO[0m[0231] Taking snapshot of full filesystem...        
[36mINFO[0m[0273] Pushing layer [MASKED]/my-project/_cache:c974dabca35fa1f807caf385eafd8d8b9951c1e81a2cad5f18630ecd3fd7d00b to cache now 
[36mINFO[0m[0273] Pushing image to [MASKED]/my-project/_cache:c974dabca35fa1f807caf385eafd8d8b9951c1e81a2cad5f18630ecd3fd7d00b 

The content of the manifest.json for the cache layer c974dabca35fa1f807caf385eafd8d8b9951c1e81a2cad5f18630ecd3fd7d00b is:

{
	"schemaVersion": 2,
	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
	"config": {
		"mediaType": "application/vnd.docker.container.image.v1+json",
		"size": 321,
		"digest": "sha256:a4a182eb882a11689b347371c2af54a740a420c0bcbd330d106c33065a7e1c95"
	},
	"layers": [
		{
			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
			"size": 721161501,
			"digest": "sha256:7f6676bd0a9c6e93f63085cf1190056047bc3e3dfbee7fb8da34b35487f2d4f2"
		}
	]
}

However the sha256_7f6676bd0a9c6e93f63085cf1190056047bc3e3dfbee7fb8da34b35487f2d4f2 file in Artifactory has indeed 0 bytes, explaining why in a subsequent build I run into this error message:

INFO[0001] Checking for cached layer [MASKED]/my-project/_cache:db5dfc4211abf7e17354bfd97ecf6b1848fdd7070724f14a231abcef637e2266... 
INFO[0001] Using caching version of cmd: COPY ./docker/development/install_application_dependencies.sh /tmp/ 
INFO[0001] Checking for cached layer [MASKED]/my-project/_cache:c974dabca35fa1f807caf385eafd8d8b9951c1e81a2cad5f18630ecd3fd7d00b... 
INFO[0001] Using caching version of cmd: RUN /tmp/install_application_dependencies.sh 
INFO[0001] Skipping unpacking as no commands require it. 
INFO[0001] ENV PYTHON_VERSION=3.6                       
INFO[0001] No files changed in this command, skipping snapshotting. 
INFO[0001] ENV LLVM_CONFIG=/usr/bin/llvm-config-10      
INFO[0001] No files changed in this command, skipping snapshotting. 
INFO[0001] ENV TZ=Europe/Berlin                         
INFO[0001] No files changed in this command, skipping snapshotting. 
INFO[0001] COPY ./docker/development/install_application_dependencies.sh /tmp/ 
INFO[0001] Found cached layer, extracting to filesystem 
INFO[0001] RUN /tmp/install_application_dependencies.sh 
INFO[0001] Found cached layer, extracting to filesystem 
error building image: error building stage: failed to execute command: extracting fs from image: error verifying size; got 0, want 721161501

Expected behavior

Kaniko should upload a file with the expected 721161501 bytes, not 0 bytes.

To Reproduce

I can consistently reproduce this behavior. The Dockerfile that causes this issue contains this (I removed anything after the offending layer):

ARG BASE_IMAGE
FROM ${BASE_IMAGE} AS base

ENV PYTHON_VERSION=3.6
ENV LLVM_CONFIG=/usr/bin/llvm-config-10
ENV TZ=Europe/Berlin

COPY ./docker/development/install_application_dependencies.sh /tmp/
RUN /tmp/install_application_dependencies.sh

The content of install_application_dependencies.sh is

set -euxo pipefail
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get -y install dnsutils netcat
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash
apt-get -y install git-lfs
apt install -y libxml2-dev libxslt-dev

Additional Information

• Dockerfile: see "To reproduce"
• Build Context: see "To reproduce"
• Kaniko Image (fully qualified with digest): sha256:e1e8f684a8d2023df1c21f06f2a66f7ec7ef965e6d092c7efe00ca6d592b2e6b from gcr.io/kaniko-project/executor:debug

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [ ]
Please check if this error is seen when you use --cache flag	- [x]
Please check if your dockerfile is a multistage dockerfile	- [x]

[NiklasRosenstein]

I changed the base image from ubuntu:bionic to ubuntu:focal and I don't see this issue anymore.