build step on multistep task fails when cache:enabled and base image in private ACR

[infbase]

Describe the bug On a multistep task, when cache:enabled is set on a build step and base image is in a private ACR, task fails when fetching base image digest.

2021/08/13 19:58:09 Downloading source code...
2021/08/13 19:58:11 Finished downloading source code
2021/08/13 19:58:11 Alias support enabled for version >= 1.1.0, please see https://aka.ms/acr/tasks/task-aliases for more information.
2021/08/13 19:58:14 Creating Docker network: acb_default_network, driver: 'bridge'
2021/08/13 19:58:14 Successfully set up Docker network: acb_default_network
2021/08/13 19:58:14 Setting up Docker configuration...
2021/08/13 19:58:15 Successfully set up Docker configuration
2021/08/13 19:58:15 Logging in to registry: cscm1nacr002.azurecr.io
2021/08/13 19:58:16 Successfully logged into cscm1nacr002.azurecr.io
2021/08/13 19:58:16 Task will use build cache, initializing buildkitd container
(...)
2021/08/13 19:58:16 Successfully executed container: acrbuildkitdcontainer
(...)
2021/08/13 19:58:16 Successfully executed container: login-azure
(...)
2021/08/13 19:58:16 Successfully executed container: login-cscm1hacr001
(...)
2021/08/13 19:58:27 Successfully executed container: git-pull
(...)
2021/08/13 19:58:28 Launching container with name: img-build
#1 [internal] booting buildkit
#1 sha256:ab77d68d70b1ef1f824aba9b5527c5e90a4179e2dd223c2510e7be1faa33db71
#1 checking if image 'buildkit' exists locally done
#1 creating container buildx_buildkit_interesting_almeida0
#1 creating container buildx_buildkit_interesting_almeida0 0.9s done
#1 DONE 0.9s
(...)
#7 [final 1/6] FROM cscm1hacr001.azurecr.io/it-ccoe/aspnet:5.0-dd@sha256:fc460fe7db0a51a32a4f0df459a8c7e87d4dba437d5a6b20fd0d48adc92dc3ea
#7 sha256:671cd6a1d8dd798ad36dc3b1f9ec0baf0cac9e8962614403e9e71cc52e89d01f
#7 resolve cscm1hacr001.azurecr.io/it-ccoe/aspnet:5.0-dd@sha256:fc460fe7db0a51a32a4f0df459a8c7e87d4dba437d5a6b20fd0d48adc92dc3ea 0.0s done
#7 DONE 0.0s
(...)
#26 exporting cache
#26 sha256:2700d4ef94dee473593c5c614b55b2dedcca7893909811a8f2b48291a1f581e4
#26 preparing build cache for export done
(...)
#26 DONE 1.2s
2021/08/13 19:59:01 Successfully executed container: img-build
021/08/13 19:59:01 Step ID: login-azure marked as successful (elapsed time in seconds: 4.408729)
2021/08/13 19:59:01 Step ID: login-cscm1hacr001 marked as successful (elapsed time in seconds: 6.722074)
2021/08/13 19:59:01 Step ID: git-pull marked as successful (elapsed time in seconds: 4.420412)
2021/08/13 19:59:01 Step ID: img-build marked as successful (elapsed time in seconds: 34.019768)
2021/08/13 19:59:01 Populating digests for step ID: img-build...
2021/08/13 19:59:01 Image was built using buildkit, fetching Digest from remote...
Failed to Resolve the reference 'cscm1hacr001.azurecr.io/it-ccoe/aspnet:5.0-dd': failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized
Run ID: cb84 failed after 54s. Error: failed during run, err: exit status 1

If az acr task credential add is used to persist login credentials for the other private ACR, behavior is the same as issue #566

To Reproduce Steps to reproduce the behavior:

Multistep task definition:

version: v1.1.0
secrets:
  - id: username
    keyvault: https://...
  - id: password
    keyvault: https://...
steps:
  - cmd: az login --identity
    id: login-azure
    when: ["-"]
  - cmd: docker login cscm1hacr001.azurecr.io --username {{.Secrets.username}} --password {{.Secrets.password}}
    id: login-cscm1hacr001
    retries: 3
    retryDelay: 10
    when: ["-"]
  - cmd: >
      cscm1hacr001.azurecr.io/it-ccoe/imgbuilder:1.0
      mkdir git && cd git
      && git -c http.extraHeader="Authorization: Basic {{.Values.ADO_PAT}}" clone {{.Values.ADO_URL}} --branch {{.Values.ADO_BRANCH}} --single-branch .
      && git -c http.extraHeader="Authorization: Basic {{.Values.ADO_PAT}}" reset {{.Values.ADO_COMMIT}} --hard
    id: git-pull
    retries: 3
    retryDelay: 20
    pull: false
    when: ["login-cscm1hacr001"]
  - build: > 
      --tag {{.Run.Registry}}/{{.Values.REPO_NAME}}:{{.Run.ID}} 
      --file /workspace/git/{{.Values.BUILD_PATH}}/{{.Values.BUILD_FILE}}
      /workspace/git/{{.Values.BUILD_PATH}}
    id: img-build
    cache: enabled
    when: ["login-cscm1hacr001", "git-pull"]

Multistep task definition published as builder-task.yaml to oci://cscm1nacr002.azurecr.io/it-alm/builder-task:1.0

Multistep task usage:

Param(
    [Parameter(Mandatory=$true)][string]$gitToken,
    [string]$adoURL = "https://dev.azure.com/...",
    [string]$adoBranch = "...",
    [string]$adoCommit = "...",
    [string]$acrName = "cscm1nacr002",
    [string]$repoName = "...",
    [string]$platform = "Linux",
    [string]$buildPath = "...",
    [string]$buildFile = "Dockerfile",
    [bool]$requiresInternalVNet = $false
)

$managedIdentity = "<resource id>"
$adoPAT = [Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes(":$gitToken"))
$taskName = ($repoName.replace("/","-") + "-${adoBranch}-${adoCommit}").substring(0,50)

az config set --local defaults.acr=$acrName

$acrTaskCmd = {
az acr task create `
  --output json `
  --name $taskName `
  --image $repoName `
  --platform $platform `
  --context "oci://cscm1nacr002.azurecr.io/it-alm/builder-task:1.0" `
  --file "builder-task.yaml" `
  --commit-trigger-enabled false `
  --pull-request-trigger-enabled false `
  --base-image-trigger-enabled true `
  --base-image-trigger-type "All" `
  --assign-identity $managedIdentity `
  --set ADO_PAT=$adoPAT `
  --set ADO_URL=$adoURL `
  --set ADO_BRANCH=$adoBranch `
  --set ADO_COMMIT=$adoCommit `
  --set REPO_NAME=$repoName `
  --set BUILD_PATH=$buildPath `
  --set BUILD_FILE=$buildFile `
}

if($requiresInternalVNet) {
  $acrTaskCmd = [ScriptBlock]::Create($acrTaskCmd.ToString() + "--agent-pool ${acrName}-APDH01")
}

& $acrTaskCmd

az acr task run -n $taskName

Expected behavior The build step should be able to successfully create and push the cache.

[Wwwsylvia]

@northtyphoon Can Tasks folks take a look at this?

[terencet-dev]

Closing as this has been inactive for over three months. Please open a support ticket with our team for assistance.