kaniko icon indicating copy to clipboard operation
kaniko copied to clipboard
verified publisher icon GoogleContainerTools

Snapshotting is taking > 10 minutes

Open nicolaferraro opened this issue 5 years ago • 28 comments

Actual behavior

We're using Kaniko in Camel K and some users are complaining about builds that became slow after we upgraded to v0.14.0 from 0.9.0 (https://github.com/apache/camel-k/issues/1209) especially on Minikube (towards Minikube internal addon registry).

Under the hood, we use the fabric8/s2i-java:3.0-java8 image which is a bit heavy, but it seems the logs get stuck when doing the snapshot of some jar files in a specific dir (~50 jars):

camel-k-kit-bofk6gui1aal47gim840-builder kaniko INFO[0492] Downloading base image 10.111.197.108/chapter-7/camel-k-kit-bofj39ei1aal47gim830:36212
camel-k-kit-bofk6gui1aal47gim840-builder kaniko INFO[0647] Skipping unpacking as no commands require it.
camel-k-kit-bofk6gui1aal47gim840-builder kaniko INFO[0647] Taking snapshot of full filesystem...
camel-k-kit-bofk6gui1aal47gim840-builder kaniko INFO[0647] Using files from context: [/builder/kit-bofk6gui1aal47gim840/package/context]
camel-k-kit-bofk6gui1aal47gim840-builder kaniko INFO[0647] ADD . /deployments
camel-k-kit-bofk6gui1aal47gim840-builder kaniko INFO[0647] Taking snapshot of files...

Is there any way to speedup Kaniko builds, e.g. by installing something on the host machine? Shouldn't the snapshot be limited to the added directory after an "ADD" command?

Expected behavior It should not take ages to build an image.

To Reproduce Steps to reproduce the behavior:

  1. Install Minikube
  2. Enable the registry addon
  3. Install Camel K 1.0.0-RC1 (kamel install)
  4. Build an integration kamel run https://raw.githubusercontent.com/apache/camel-k/master/examples/simple.groovy
  5. A kaniko build pod will be started, preceded by an init-container

Triage Notes for the Maintainers

Description Yes/No
Please check if this a new feature you are proposing
  • - [ ]
Please check if the build works in docker but not in kaniko
  • - [ ]
Please check if this error is seen when you use --cache flag
  • - [ ]
Please check if your dockerfile is a multistage dockerfile
  • - [ ]

nicolaferraro avatar Jan 16 '20 12:01 nicolaferraro

@nicolaferraro For ADD command we only take snapshot of files the log line below also suggests it.

camel-k-kit-bofk6gui1aal47gim840-builder kaniko INFO[0647] Taking snapshot of files...

https://github.com/GoogleContainerTools/kaniko/blob/master/pkg/snapshot/snapshot.go#L63

Can you provide output with -v=debug flag? Maybe the number of files to be snapshotted are not just files from ADD . ./deployments dir .

tejal29 avatar Jan 17 '20 06:01 tejal29

Also, can you provide your dockerfile? Are you copying the / ? see #960

tejal29 avatar Jan 17 '20 06:01 tejal29

Hi @tejal29, the Dockerfile is simple:

FROM fabric8/s2i-java:3.0-java8
ADD . /deployments

I've extracted a verbose log. The problem is that after the snapshot there are no more logs, so it's difficult to tell what's going slow.

camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0000] Copying file /builder/kit-boin6figbopchis5erg0/package/context/Dockerfile to /kaniko/Dockerfile 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0000] Skip resolving path /kaniko/Dockerfile       
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0000] Skip resolving path /builder/kit-boin6figbopchis5erg0/package/context 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0000] Skip resolving path /kaniko/cache            
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0000] Skip resolving path                          
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0000] Skip resolving path                          
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0000] Skip resolving path                          
camel-k-kit-boin6figbopchis5erg0-builder kaniko INFO[0000] Resolved base name fabric8/s2i-java:3.0-java8 to fabric8/s2i-java:3.0-java8 
camel-k-kit-boin6figbopchis5erg0-builder kaniko INFO[0000] Resolved base name fabric8/s2i-java:3.0-java8 to fabric8/s2i-java:3.0-java8 
camel-k-kit-boin6figbopchis5erg0-builder kaniko INFO[0000] Downloading base image fabric8/s2i-java:3.0-java8 
camel-k-kit-boin6figbopchis5erg0-builder kaniko INFO[0003] Error while retrieving image from cache: getting file info: stat /kaniko/cache/sha256:9ac5acb3aa8e7e179eb1fd28cda7dac22ef7deac24f00b50cee56265fa259141: no such file or directory 
camel-k-kit-boin6figbopchis5erg0-builder kaniko INFO[0003] Downloading base image fabric8/s2i-java:3.0-java8 
camel-k-kit-boin6figbopchis5erg0-builder kaniko INFO[0005] Built cross stage deps: map[]                
camel-k-kit-boin6figbopchis5erg0-builder kaniko INFO[0005] Downloading base image fabric8/s2i-java:3.0-java8 
camel-k-kit-boin6figbopchis5erg0-builder kaniko INFO[0006] Error while retrieving image from cache: getting file info: stat /kaniko/cache/sha256:9ac5acb3aa8e7e179eb1fd28cda7dac22ef7deac24f00b50cee56265fa259141: no such file or directory 
camel-k-kit-boin6figbopchis5erg0-builder kaniko INFO[0006] Downloading base image fabric8/s2i-java:3.0-java8 
camel-k-kit-boin6figbopchis5erg0-builder kaniko INFO[0008] Executing 0 build triggers                   
camel-k-kit-boin6figbopchis5erg0-builder kaniko INFO[0008] Skipping unpacking as no commands require it. 
camel-k-kit-boin6figbopchis5erg0-builder kaniko INFO[0008] Taking snapshot of full filesystem...        
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Skipping paths under /kaniko, as it is a whitelisted directory 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Skipping paths under /var/run, as it is a whitelisted directory 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Skipping paths under /builder, as it is a whitelisted directory 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Skipping paths under /dev, as it is a whitelisted directory 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Skipping paths under /proc, as it is a whitelisted directory 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Skipping paths under /sys, as it is a whitelisted directory 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Adding / to layer, because it was changed.   
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Adding /var to layer, because it was changed. 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Adding /etc to layer, because it was changed. 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Adding /.dockerenv to layer, because it was changed. 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Adding /workspace to layer, because it was changed. 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Resolved . to .                              
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Resolved /deployments to /deployments        
camel-k-kit-boin6figbopchis5erg0-builder kaniko INFO[0008] Using files from context: [/builder/kit-boin6figbopchis5erg0/package/context] 
camel-k-kit-boin6figbopchis5erg0-builder kaniko INFO[0008] ADD . /deployments                           
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Resolved . to .                              
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Resolved /deployments to /deployments        
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Resolved . to .                              
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Resolved /deployments to /deployments        
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Getting files and contents at root /builder/kit-boin6figbopchis5erg0/package/context 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Creating directory /deployments              
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/Dockerfile to /deployments/Dockerfile 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Creating directory /deployments/dependencies 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/com.sun.xml.bind.jaxb-core-2.3.0.1.jar to /deployments/dependencies/com.sun.xml.bind.jaxb-core-2.3.0.1.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/com.sun.xml.bind.jaxb-impl-2.3.0.jar to /deployments/dependencies/com.sun.xml.bind.jaxb-impl-2.3.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/commons-io.commons-io-2.6.jar to /deployments/dependencies/commons-io.commons-io-2.6.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/javax.xml.bind.jaxb-api-2.3.0.jar to /deployments/dependencies/javax.xml.bind.jaxb-api-2.3.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.camel-api-3.0.0.jar to /deployments/dependencies/org.apache.camel.camel-api-3.0.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.camel-base-3.0.0.jar to /deployments/dependencies/org.apache.camel.camel-base-3.0.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.camel-bean-3.0.0.jar to /deployments/dependencies/org.apache.camel.camel-bean-3.0.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.camel-core-engine-3.0.0.jar to /deployments/dependencies/org.apache.camel.camel-core-engine-3.0.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.camel-endpointdsl-3.0.0.jar to /deployments/dependencies/org.apache.camel.camel-endpointdsl-3.0.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.camel-groovy-3.0.0.jar to /deployments/dependencies/org.apache.camel.camel-groovy-3.0.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.camel-jaxp-3.0.0.jar to /deployments/dependencies/org.apache.camel.camel-jaxp-3.0.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.camel-log-3.0.0.jar to /deployments/dependencies/org.apache.camel.camel-log-3.0.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.camel-main-3.0.0.jar to /deployments/dependencies/org.apache.camel.camel-main-3.0.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.camel-management-api-3.0.0.jar to /deployments/dependencies/org.apache.camel.camel-management-api-3.0.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.camel-support-3.0.0.jar to /deployments/dependencies/org.apache.camel.camel-support-3.0.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.camel-timer-3.0.0.jar to /deployments/dependencies/org.apache.camel.camel-timer-3.0.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.camel-util-3.0.0.jar to /deployments/dependencies/org.apache.camel.camel-util-3.0.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.camel-util-json-3.0.0.jar to /deployments/dependencies/org.apache.camel.camel-util-json-3.0.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.k.camel-k-loader-groovy-1.0.9.jar to /deployments/dependencies/org.apache.camel.k.camel-k-loader-groovy-1.0.9.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.k.camel-k-runtime-core-1.0.9.jar to /deployments/dependencies/org.apache.camel.k.camel-k-runtime-core-1.0.9.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.k.camel-k-runtime-main-1.0.9.jar to /deployments/dependencies/org.apache.camel.k.camel-k-runtime-main-1.0.9.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.camel.spi-annotations-3.0.0.jar to /deployments/dependencies/org.apache.camel.spi-annotations-3.0.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.commons.commons-lang3-3.9.jar to /deployments/dependencies/org.apache.commons.commons-lang3-3.9.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.logging.log4j.log4j-api-2.13.0.jar to /deployments/dependencies/org.apache.logging.log4j.log4j-api-2.13.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.logging.log4j.log4j-core-2.13.0.jar to /deployments/dependencies/org.apache.logging.log4j.log4j-core-2.13.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.apache.logging.log4j.log4j-slf4j-impl-2.13.0.jar to /deployments/dependencies/org.apache.logging.log4j.log4j-slf4j-impl-2.13.0.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.codehaus.groovy.groovy-2.5.8.jar to /deployments/dependencies/org.codehaus.groovy.groovy-2.5.8.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Copying file /builder/kit-boin6figbopchis5erg0/package/context/dependencies/org.slf4j.slf4j-api-1.7.29.jar to /deployments/dependencies/org.slf4j.slf4j-api-1.7.29.jar 
camel-k-kit-boin6figbopchis5erg0-builder kaniko INFO[0008] Taking snapshot of files...                  
camel-k-kit-boin6figbopchis5erg0-builder kaniko DEBU[0008] Taking snapshot of files [/deployments /deployments/Dockerfile /deployments/dependencies /deployments/dependencies/com.sun.xml.bind.jaxb-core-2.3.0.1.jar /deployments/dependencies/com.sun.xml.bind.jaxb-impl-2.3.0.jar /deployments/dependencies/commons-io.commons-io-2.6.jar /deployments/dependencies/javax.xml.bind.jaxb-api-2.3.0.jar /deployments/dependencies/org.apache.camel.camel-api-3.0.0.jar /deployments/dependencies/org.apache.camel.camel-base-3.0.0.jar /deployments/dependencies/org.apache.camel.camel-bean-3.0.0.jar /deployments/dependencies/org.apache.camel.camel-core-engine-3.0.0.jar /deployments/dependencies/org.apache.camel.camel-endpointdsl-3.0.0.jar /deployments/dependencies/org.apache.camel.camel-groovy-3.0.0.jar /deployments/dependencies/org.apache.camel.camel-jaxp-3.0.0.jar /deployments/dependencies/org.apache.camel.camel-log-3.0.0.jar /deployments/dependencies/org.apache.camel.camel-main-3.0.0.jar /deployments/dependencies/org.apache.camel.camel-management-api-3.0.0.jar /deployments/dependencies/org.apache.camel.camel-support-3.0.0.jar /deployments/dependencies/org.apache.camel.camel-timer-3.0.0.jar /deployments/dependencies/org.apache.camel.camel-util-3.0.0.jar /deployments/dependencies/org.apache.camel.camel-util-json-3.0.0.jar /deployments/dependencies/org.apache.camel.k.camel-k-loader-groovy-1.0.9.jar /deployments/dependencies/org.apache.camel.k.camel-k-runtime-core-1.0.9.jar /deployments/dependencies/org.apache.camel.k.camel-k-runtime-main-1.0.9.jar /deployments/dependencies/org.apache.camel.spi-annotations-3.0.0.jar /deployments/dependencies/org.apache.commons.commons-lang3-3.9.jar /deployments/dependencies/org.apache.logging.log4j.log4j-api-2.13.0.jar /deployments/dependencies/org.apache.logging.log4j.log4j-core-2.13.0.jar /deployments/dependencies/org.apache.logging.log4j.log4j-slf4j-impl-2.13.0.jar /deployments/dependencies/org.codehaus.groovy.groovy-2.5.8.jar /deployments/dependencies/org.slf4j.slf4j-api-1.7.29.jar]

nicolaferraro avatar Jan 20 '20 09:01 nicolaferraro

"Taking snapshot of full filesystem" is taking > 30 minutes.. Can't start using kaniko because of this :(

My Dockerfile is very simple:

FROM node:10 as builder
WORKDIR /usr/src/app
COPY . .
ENV NODE_ENV=production
RUN yarn install --frozen-lockfile --production; \
        yarn build

DjinNO avatar Jan 22 '20 08:01 DjinNO

Let me look in to it @DjinNO Can you also specify what is the size of your current workdir?

tejal29 avatar Jan 22 '20 23:01 tejal29

Thanks @nicolaferraro Can you change your dockerfile to

FROM fabric8/s2i-java:3.0-java8
ADD /deployments /deployments

and see if it helps? I want able to reproduce your error. i will try downloading one of these jars in my workspace and see if i can reproduce this issue.

tejal29 avatar Jan 23 '20 00:01 tejal29

@nicolaferraro i am not able to reproduce this issue on my end. I downloaded a jar and put it in the my $buildcontext/dependencies directory

DEBU[0013] Taking snapshot of files [/deployments .... /deployments/dependencies/org.apache.camel.camel-util-3.0.0.jar 
...
 /integration_test.go /deployments/replace-gcr-with-local-registry.sh] 

DEBU[0013] build: composite key for command ADD . /deployments &{[sha256:9ac5acb3aa8e7e179eb1fd28cda7dac22ef7deac24f00b50cee56265fa259141 ADD . /deployments 9c3ae8419cd2a88d7f7bfce76d25d10c5836b95282d8e632bb17697cc4b9b764]} 
DEBU[0013] build: cache key for command ADD . /deployments 4b5f61364d4092a8866f0e202e155ac62a8dc5a3278a929e1195d918d5e14e81 
DEBU[0018] mapping stage idx 0 to digest sha256:2a5f2fcb115dc8acf1c7eb46c2df112a3983950ffd58e28b962c6dd7b2f547a5

Can you provide more information? Are any of these jars symlinks?

tejal29 avatar Jan 23 '20 00:01 tejal29

@nicolaferraro i pushed an image gcr.io/kaniko-project/executor:970 with fix #989. I want to know if we spend to much time calculating the parent directories. Can you please try this kaniko image?

Thanks Tejal

tejal29 avatar Jan 23 '20 01:01 tejal29

Can you also specify what is the size of your current workdir?

80M

DjinNO avatar Jan 23 '20 08:01 DjinNO

Im using gitlab ci to build image and i want to store cache in gitlab registry. To try to use gcr.io/kaniko-project/executor:970 i need shell in image

DjinNO avatar Jan 23 '20 08:01 DjinNO

@DjinNO ok,I pushed a debug image.

gcr.io/kaniko-project/executor:debug_970

tejal29 avatar Jan 23 '20 19:01 tejal29

@DjinNO ok,I pushed a debug image.

gcr.io/kaniko-project/executor:debug_970

With this image i have error: error building image: error building stage: unable to add file to layered map: error creating hash for : lstat : no such file or directory

On first snapshot

DjinNO avatar Jan 24 '20 07:01 DjinNO

I've created #1013 that is a follow-up analysis of the build time for Apache Camel K.

astefanutti avatar Jan 30 '20 09:01 astefanutti

Thanks @astefanutti, yes snapshotting does not seem to be the problem, it's only referenced in last line printed in the logs. Kaniko takes a lot to push the image to the registry.

nicolaferraro avatar Jan 30 '20 11:01 nicolaferraro

I seem to have a similar problem with some private builds. I am a JavaScript dev, so I have yarn and node environments.

Using kaniko debug-v0.19.0, I get stuck in the following:

INFO[0547] Taking snapshot of full filesystem...        
INFO[0548] Resolving paths

The process seems to be stuck and it doesn't use any CPU (CPU dropped from 1 to 0.06).

Some things I noticed in my private build

  • yarn cache dir could be whitelisted?, although I don't think that can be done in kaniko because its not technically needed for the actual build.

  • Single snapshot helps.

hrgui avatar Apr 10 '20 23:04 hrgui

This is still happening for me. I do have a huge container; however, sometimes its quick taking the snapshot other times its not.

kylecribbs avatar Jun 08 '20 20:06 kylecribbs

@kylecribbs the later versions of kaniko explain more to why as it says its snapshotting X directories. What version of kaniko are you using?

hrgui avatar Jun 09 '20 01:06 hrgui

I'm also experiencing this in cloud build... the image is large (the cached layers are 400mb in container registry), but every Full Snapshot after installing node_modules takes many minutes.

Shouldn't subsequent layers be small?

Is there a debug image and some parameters you'd like me to use to get a log?

I'm not sure if it's the snapshotting or the pushing that's taking a long time.

martaver avatar Sep 03 '20 15:09 martaver

I'm also in europe-north1... I know it's a long shot but... does it matter which region your container registry is in?

martaver avatar Sep 03 '20 15:09 martaver

The layers that are being pushed by the long-running 'full snapshot' steps end up being quite small... 2MB, 100 bytes etc...

That leads me to believe that it's the actual snapshot step that's slow and long-running. Maybe something with the file system.

A combination of many files (node_modules) and possibly symlinks? I'm using yarn workspaces...

martaver avatar Sep 04 '20 05:09 martaver

I'm having the same problem, 40+ minutes makes it unusable. :(

cope avatar Feb 26 '21 15:02 cope

I am having the same problem, sometimes is taking just 10 minutes, but in another cases is taking more than 20 minutes

IgnacioYanjari avatar Mar 31 '21 22:03 IgnacioYanjari

Same problem here 25+ minutes, a lot of layers of 2MB size I don't know if it has to be with my node_modules folder.

dranes avatar May 10 '21 15:05 dranes

Was this issue fixed?

kerryjiang avatar Jul 30 '21 17:07 kerryjiang

@jonjohnsonjr I just had a build with the latest version of Kaniko that took 20min to build and timeouted after 40m of "taking snapshot. Reopen the issue. It's a serious one. Obviously something was overlooked here.

And I've tried recommended advice such as: --use-new-run and --snapshotMode=redo.

itay-grudev avatar Jan 05 '22 10:01 itay-grudev

@itay-grudev Thanks for trying out the latest Kaniko, this feedback is super helpful to avoid pushing these changes out to all users.

Do you happen to have a Dockerfile that can reproduce this slow snapshotting issue? It would be immensely helpful in debugging the problem, and preventing future regressions.

It looks like the fix in #1002 indicates this slowness might come from pushing to an insecure registry, is that possibly related to the slowness you're seeing?

imjasonh avatar Jan 05 '22 15:01 imjasonh

@imjasonh Here is some more information if that could be useful - my job that triggers a Kaniko build:

build:
  image:
    name: gcr.io/kaniko-project/executor:v1.7.0-debug
    entrypoint: [""]
  retry: 2
  variables:
    GIT_STRATEGY: fetch
    # More Information on Kaniko Caching: https://cloud.google.com/build/docs/kaniko-cache
    KANIKO_CACHE_ARGS: "--cache=true --cache-copy-layers=true --cache-ttl=96h"
    VERSIONLABELMETHOD: "OnlyIfThisCommitHasVersion" # options: "OnlyIfThisCommitHasVersion","LastVersionTagInGit"
    IMAGE_LABELS: >
      --label org.opencontainers.image.vendor=$CI_SERVER_URL/$GITLAB_USER_LOGIN
      --label org.opencontainers.image.authors=$CI_SERVER_URL/$GITLAB_USER_LOGIN
      --label org.opencontainers.image.revision=$CI_COMMIT_SHA
      --label org.opencontainers.image.source=$CI_PROJECT_URL
      --label org.opencontainers.image.documentation=$CI_PROJECT_URL
      --label org.opencontainers.image.licenses=$CI_PROJECT_URL
      --label org.opencontainers.image.url=$CI_PROJECT_URL
      --label vcs-url=$CI_PROJECT_URL
      --label com.gitlab.ci.user=$CI_SERVER_URL/$GITLAB_USER_LOGIN
      --label com.gitlab.ci.email=$GITLAB_USER_EMAIL
      --label com.gitlab.ci.tagorbranch=$CI_COMMIT_REF_NAME
      --label com.gitlab.ci.pipelineurl=$CI_PIPELINE_URL
      --label com.gitlab.ci.commiturl=$CI_PROJECT_URL/commit/$CI_COMMIT_SHA
      --label com.gitlab.ci.cijoburl=$CI_JOB_URL
      --label com.gitlab.ci.mrurl=$CI_PROJECT_URL/-/merge_requests/$CI_MERGE_REQUEST_ID
  script:
    - |
      echo "Building and shipping image to $CI_REGISTRY_IMAGE"
      # Build date for opencontainers
      BUILDDATE="'$(date '+%FT%T%z' | sed -E -n 's/(\+[0-9]{2})([0-9]{2})$/\1:\2/p')'" #rfc 3339 date
      IMAGE_LABELS="$IMAGE_LABELS --label org.opencontainers.image.created=$BUILDDATE --label build-date=$BUILDDATE"
      #Description for opencontainers
      BUILDTITLE=$(echo $CI_PROJECT_TITLE | tr " " "_")
      IMAGE_LABELS="$IMAGE_LABELS --label org.opencontainers.image.title=$BUILDTITLE --label org.opencontainers.image.description=$BUILDTITLE"
      #Add ref.name for opencontainers
      IMAGE_LABELS="$IMAGE_LABELS --label org.opencontainers.image.ref.name=$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME"

      # Build Version Label and Tag from git tag, LastVersionTagInGit was placed by a previous job artifact
      if [[ "$VERSIONLABELMETHOD" == "LastVersionTagInGit" ]]; then VERSIONLABEL=$(cat VERSIONTAG.txt); fi
      if [[ "$VERSIONLABELMETHOD" == "OnlyIfThisCommitHasVersion" ]]; then VERSIONLABEL=$CI_COMMIT_TAG; fi
      if [[ ! -z "$VERSIONLABEL" ]]; then
        IMAGE_LABELS="$IMAGE_LABELS --label org.opencontainers.image.version=$VERSIONLABEL"
        ADDITIONALTAGLIST="$ADDITIONALTAGLIST $VERSIONLABEL"
      fi

      ADDITIONALTAGLIST="$ADDITIONALTAGLIST $CI_COMMIT_REF_NAME $CI_COMMIT_SHORT_SHA"
      if [[ "$CI_COMMIT_BRANCH" == "$CI_DEFAULT_BRANCH" ]]; then ADDITIONALTAGLIST="$ADDITIONALTAGLIST latest"; fi
      if [[ -n "$ADDITIONALTAGLIST" ]]; then
        for TAG in $ADDITIONALTAGLIST; do
          FORMATTEDTAGLIST="${FORMATTEDTAGLIST} --tag $CI_REGISTRY_IMAGE:$TAG ";
        done;
      fi

      # Reformat Docker tags to kaniko's --destination argument:
      FORMATTEDTAGLIST=$(echo "${FORMATTEDTAGLIST}" | sed s/\-\-tag/\-\-destination/g)

      mkdir -p /kaniko/.docker
      echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n $CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD | base64)\"}}}" > /kaniko/.docker/config.json
      /kaniko/executor --context $CI_PROJECT_DIR \
        --dockerfile $CI_PROJECT_DIR/Dockerfile \
        --build-arg RAILS_MASTER_KEY=$RAILS_MASTER_KEY \
        --use-new-run \
        $KANIKO_CACHE_ARGS \
        $FORMATTEDTAGLIST \
        $IMAGE_LABELS

itay-grudev avatar Jan 10 '22 10:01 itay-grudev

There are multiple reports of this behavior on both Gitlab and Github ci pipelines. The error is rarely 100% reproducible! I have a pipeline that suddenly started getting stuck near half the time when running in our cluster but seems to work 100% on the gitlab shared runners. Unfortunately gitlab-ci doesn't let me store artifacts on timeout (doh!) and the kaniko debug output too large for the normal build log. So currently I can't provide info. If anyone wants to run some trials I can reliably fire up different kaniko versions and get a failure in minutes.

So either it takes less than 30 seconds on success, or it seems to go on forever (never tested more than ~25 minutes).

I've tried a plethora of different options but nothing seems to have the slightest effect.

Joen-UnLogick avatar Jun 07 '22 09:06 Joen-UnLogick

We use kaniko version 1.9.1 and have same problem. Very long BuildImage. Especially long time take push stage. According job log, push stage takes more than 10 minutes:

` DEBU[0366] mapping digest sha256:8cbfdebd35cd3cfa982544b61a82d7f23c6c23ad29a157773a00aa7 to cachekey

INFO[0366] Pushing image to 1.8.1.1/some/image:0.1.0.437

INFO[1146] Pushed 1.8.1.1/some/image@sha256:14018e25fc51f00dbe21a473861749a78ac28d284087cff252265d2 ` From 0366 seconds to 1146.

Homicevich avatar Nov 03 '22 06:11 Homicevich

I have the same issue inside Gitlab on a runner with 128 GB RAM and 16 cores. Using --snapshot-mode=redo and --use-new-run yields no performance improvements.

For a 468 MB image the actual building takes 4-5 seconds yet the snapshotting and unpacking rootfs take 55 seconds. I think this is unacceptable and very slow:

$ executor --context ./service --dockerfile ./service/Dockerfile --build-arg 'COMMIT_SHA=REDACTED' \
--label 'org.opencontainers.image.source=REDACTED' \
--label 'org.opencontainers.image.created=2022-11-09 12:09:55' \
--destination=image --snapshot-mode=redo --use-new-run --tarPath oci-image.tar --no-push

INFO[0000] Retrieving image manifest $REDACTED 
INFO[0000] Retrieving image $REDACTED
INFO[0000] Built cross stage deps: map[]                
INFO[0000] Retrieving image manifest $REDACTED
INFO[0000] Returning cached image manifest              
INFO[0000] Executing 0 build triggers                   
INFO[0000] Building stage $REDACTED [idx: '0', base-idx: '-1'] 
INFO[0000] Unpacking rootfs as cmd COPY run.sh run.sh requires it. 
INFO[0012] Initializing snapshotter ...                 
INFO[0012] Taking snapshot of full filesystem...        
INFO[0056] WORKDIR /app                                 
INFO[0056] Cmd: workdir                                 
INFO[0056] Changed working directory to /app            
INFO[0056] Creating directory /app                      
INFO[0056] Taking snapshot of files...                  
INFO[0057] CMD ["sh", "run.sh"]                         
INFO[0057] COPY run.sh run.sh                           
INFO[0057] Taking snapshot of files...                  
INFO[0057] COPY target/ target/                         
INFO[0057] Taking snapshot of files...                  
INFO[0059] COPY config/ config/                         
INFO[0059] Taking snapshot of files...                  
INFO[0059] COPY .deploy /.deploy                        
INFO[0059] Taking snapshot of files...                  
INFO[0059] RUN if [[ ${ENV} = "dev" ]] ; then chmod -R go+rwX /app ; else  chmod -R go+rwX /app/config; fi 
INFO[0060] Cmd: /bin/sh                                 
INFO[0060] Args: [-c if [[ ${ENV} = "dev" ]] ; then chmod -R go+rwX /app ; else  chmod -R go+rwX /app/config; fi] 
INFO[0060] Running: [/bin/sh -c if [[ ${ENV} = "dev" ]] ; then chmod -R go+rwX /app ; else  chmod -R go+rwX /app/config; fi] 
INFO[0060] Taking snapshot of files...                  
INFO[0062] Skipping push to container registry due to --no-push flag

emirb avatar Nov 09 '22 13:11 emirb