Cannot configure base images stored in ACR authenticated with AKS managed credentials

[allxiao]

We deploy kpack to AKS cluster in Azure. The AKS is configured to be able to pull images from a service ACR, using the AAD integration between AKS and ACR.

The default builder's base images are from the public repository.

• paketobuildpacks/build:base-cnb
• paketobuildpacks/run:base-cnb

However, when we tried to move this to the private service ACR, we got the following error when we build with kpack:

(Reproduce steps):

1. Create an Azure Container Registry
2. Pull the builder's base image from public repo and push to the ACR created in step 1
3. Create an Azure Kubernetes Service, and setup the integration with the above ACR
4. Install kpack to this AKS
5. Create a ClusterStack, setting the build image and run image to the ones in the ACR in step 1
6. Create a Image build using the ClusterStack
7. Check the build log and verify the errors

Build reason(s): CONFIG
CONFIG:
- resources: {}
- source: {}
+ resources:
+ limits:
+ cpu: "1"
+ memory: 1Gi
+ requests:
+ cpu: "0"
+ memory: "0"
+ source:
+ git:
+ revision: 50da8d5b30ee063f2f46f87143f758e799abd2a2
+ url: https://github.com/smile37773/sample-java-app
Loading secrets for "acr2f3f2acf4f5a4932a.azurecr.io" from secret "docker-kpack"
Error verifying read access to run image "****.azurecr.io/paketobuildpacks-run:base-cnb": UNAUTHORIZED

It seems some of the logic in the build steps are not aware of the AAD integration in AKS (either service principle based or MSI based). The kpack is able to pull the lifecycle image from the private ACR, however, the build steps cannot.

We tried to add the following environment variable to the CNB image attempting to enable the AKS credential provider in build steps, however, it didn't work.

AZURE_CONTAINER_REGISTRY_CONFIG=/etc/kubernetes/azure.json

[allxiao]

As discussed with Matthew G., it's not easy to integrate the credential provider support into the build steps to resolve this problem, as that involves changes from the upstream buildpacks, which is likely to take long cycle of discussion and followed (if possible) implementation.

We may probably consider another option, to publish the bash image with our own pipeline. The base images will be built by the internal pipeline, and published to the certified public registry. There are some requirements we need to check:

1. the project must be part of the Cloud Native Computing Foundation (CNCF). The stacks project should be part of CNCF. Correct me if I'm wrong.
2. We should have Dockerfiles to build the image from source.

Currently, the stack images are built with a Go program, which will not be applicable to this pipeline. If we have Dockerfiles for the base images, we will be able to build, publish and manage the base images in the certified public registry to mitigate the problem.

[matthewmcnew]

I think this is a duplicate of https://github.com/pivotal/kpack/issues/622.

Using managed credentials is not supported within the build environment currently and will most likely require a change to the CNB lifecycle. However, the credentials configured on an image/build secret will be mounted in the build pod and will be useable to read the run-image. So, the best workaround to use a private registry for the run image is to provide the build's service account with read credentials to the run image's registry.

[matthewmcnew]

I don't believe using your own images from a Dockerfile helps as the build-init where the error does not run in the stack image.

[matthewmcnew]

Keeping this open even though https://github.com/pivotal/kpack/pull/959 was released. As additional work may be needed for ACR.