Unable to pull custom docker image in Compute Cluster from Azure private registry

[KjellZijlemaker]

Hi,

I have created a new machine learning environment with a connected ACR (without admin keys enabled). We also have created a User assigned managed identity to enable managed identity authentication for the compute clusters. This identity has the ACRPull role assigned on the connected ACR.

However, when we try the following Python code as seen in the documentation, it does not work:

env = Environment(name="private-acr")
env.docker.base_image = "as01weuacrom4vosf3mpux7.azurecr.io/custom:v1"
env.python.user_managed_dependencies = True

We get the following error message:
AzureMLCompute job failed.
FailedPullingImage: Unable to pull docker image
	imageName: as01weuacrom4vosf3mpux7.azurecr.io/custom:v1
	error: Run docker command to pull public image failed with error: Error response from daemon: Head "https://as01weuacrom4vosf3mpux7.azurecr.io/v2/custom/manifests/v1": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.
.
	Reason: Error response from daemon: Head "https://as01weuacrom4vosf3mpux7.azurecr.io/v2/custom/manifests/v1": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.

	Info: Failed to setup runtime for job execution: Job environment preparation failed on 10.235.22.6 with err exit status 1.

We also tried to build the image in ACR, but it also does not authenticate automatically. The only way we can get this to work is when we enable ACR access keys, but this obviously is not a preferred solution.

Please not that all resources are behind a vnet, including the compute cluster which has public ip disabled.

Can you please advice what to do next, or what we can try?