Build/Render/Run Fails with Kaniko + Immutable Tags

[bribroder]

I have a project which uses kaniko to build an image remotely on a cluster. The kaniko build uses a cache repo and both repos have immutable tags configured. I use the inputDigest tagPolicy and set builds to be reproducible.

When I try to skaffold run this deployment and the tagPolicy would generate a tag which already exists, skaffold still tries to use kaniko to build the image. When kaniko pushes the image, because my repo has tag immutability configured it returns an error, kaniko fails the push, and the entire skaffold run is considered failed.

Expected behavior

Builds with kaniko should check to see if a tag exists and skip the build if so.

Actual behavior

When the image is built with kaniko, skaffold checks to see if the tag exists before starting the build, doesn't seem to find the existing tag, and it tries to build anyway.

Information

• Skaffold version: v2.10.1
• Operating system: OS X 14
• Installed via: Homebrew
• Contents of skaffold.yaml:

apiVersion: skaffold/v4beta9
kind: Config

build:
  tagPolicy:
    inputDigest: {}

  artifacts:
  - image: 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo
    kaniko:
      cache:
        repo: 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo-cache
        cacheCopyLayers: true
      reproducible: true
      skipUnusedStages: true
      snapshotMode: redo
      useNewRun: true
      verbosity: info
      whitelistVarRun: true

deploy:
  helm:
    releases:
    - name: "my-app"
      chartPath: deploy/my-app
      valuesFiles:
      - deploy/my-app/values.yaml

Steps to reproduce the behavior

1. skaffold run

$ skaffold run

Generating tags...
 - 987654321.dkr.ecr.us-east-1.amazonaws.com/my-repo -> 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo:123456789
Checking cache...
 - 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo: Not found. Building
Starting build...
Building [987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo]...
Target platforms: [linux/amd64]
Flag --whitelist-var-run has been deprecated, Please use ignore-var-run instead.
time="2024-02-14T19:10:00Z" level=warning msg="Flag --snapshotMode is deprecated. Use: --snapshot-mode"
INFO[0000] Retrieving image manifest ...
[...build happens here, finds cached layers successfully, no problems...]
INFO[0331] Pushing image to 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo:123456789 
error pushing image: failed to push to destination 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo:123456789: PUT https://987654321.dkr.ecr.us-west-1.amazonaws.com/v2/my-repo/manifests/123456789: TAG_INVALID: The image tag '123456789' already exists in the 'my-repo' repository and cannot be overwritten because the repository is immutable.

I can do a build with --dry-run and then check that the generated tag does exist before the build starts:

$ skaffold build --dry-run
Generating tags...
 - 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo -> 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo:123456789
Skipping build phase since --dry-run=true

$ aws ecr list-images --repository-name ev-chargers --filter tagStatus=TAGGED --query 'imageIds[?imageTag==`'123456789'`]' --output json
[
    {
        "imageDigest": "sha256:a1s2d3f4g5h6j7k8l9",
        "imageTag": "123456789"
    }
]

#3849 sounds like a similar issue but I suspect it's just a matter of bad log messages--my cache repo is working correctly

[bribroder]

Output from a render attempt:

$ skaffold render
failed to resolve the digest of 987654321.dkr.ecr.us-west-1.amazonaws.com/my-repo:123456789: does the image exist remotely?

$ aws ecr list-images --repository-name my-repo --filter tagStatus=TAGGED --query 'imageIds[?imageTag==`'123456789'`]' --output json
[
    {
        "imageDigest": "sha256:3881176ebbf62fced4",
        "imageTag": "123456789"
    }
]