error if kaniko takes longer than 1 hour when running on AWS IRSA

Open tooptoop4 opened this issue 1 year ago • 1 comments

Actual behavior when the kaniko step (run from a pod using AWS IRSA) takes over 1 hour it fails with this error

INFO[3716] Pushing image to redact.amazonaws.com/redact.ecr/redact:0.0.72 error pushing image: failed to push to destination redact.amazonaws.com/redact.ecr/redact:0.0.72: GET https://public.ecr.aws/v2/docker/library/python/blobs/sha256:d2c04aca259ccbbbd92a78c0452532b76b5b2812b06999bafaaae910297770a9: DENIED: Your Authorization Token is invalid.

Expected behavior Image is able to be built and pushed to ECR

To Reproduce Steps to reproduce the behavior:

I am running on EKS with IRSA and trying to push to ECR
if the kaniko command (/kaniko/executor --context /repo/redact --dockerfile /repo/redact/Dockerfile --destination=redact) takes just over 1 hour then it fails

Additional Information

Dockerfile FROM public.ecr.aws/docker/library/python:3.11.9-alpine3.20 RUN #really huge curl steps here
Kaniko Image: gcr.io/kaniko-project/executor:v1.12.0-debug

env variables: - name: AWS_EC2_METADATA_DISABLED value: "true" - name: AWS_SDK_LOAD_CONFIG value: "true" - name: AWS_WEB_IDENTITY_TOKEN_FILE value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Jun 13 '24 06:06 tooptoop4

Related: https://github.com/GoogleContainerTools/kaniko/issues/2526

Jul 16 '24 18:07 aaron-prindle