[FEATURE REQ] DefaultAzureCredential for local docker testing

[et1975]

Azure.Identity Testing code that uses DefaultAzureCredential in a container locally seems to require a lot of effort, unless one is willing to supply username/password into the environment. Creating a service principal and supplying the clientID + Secret is not much better, but also requires a whole lot of additional effort - like setting up the SP, granting the permissions that the developer account already has, etc.

There should be a way to use VS/VSCode/CLI tokens simply by mounting ~/.azure into /root/.azure of the container, unfortunately this does not work today. #12749 mentions installation of the CLI as a working solution, but I just tried this on Alpine and a) it's a hassle - installing all that stuff on Alpine is error-prone experience and takes a long time (on each build!) b) it doesn't work, as I still get the exception

SharedTokenCacheCredential authentication failed: Persistence check failed. Inspect inner exception for details ---> Azure.Identity.AuthenticationFailedException: SharedTokenCacheCredential authentication failed: Persistence check failed. Inspect inner exception for details ---> Microsoft.Identity.Client.Extensions.Msal.MsalCachePersistenceException: Persistence check failed. Inspect inner exception for details ---> System.DllNotFoundException: Unable to load shared library 'libsecret-1.so.0' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Error loading shared library liblibsecret-1.so.0: No such file or directory at Microsoft.Identity.Client.Extensions.Msal.Libsecret.secret_schema_new(String name, Int32 flags, String attribute1, Int32 attribute1Type, String attribute2, Int32 attribute2Type, IntPtr end) at Microsoft.Identity.Client.Extensions.Msal.LinuxKeyringAccessor.GetLibsecretSchema() at Microsoft.Identity.Client.Extensions.Msal.LinuxKeyringAccessor.Write(Byte[] data) at Microsoft.Identity.Client.Extensions.Msal.MsalCacheStorage.VerifyPersistence() --- End of inner exception stack trace --- at Microsoft.Identity.Client.Extensions.Msal.MsalCacheStorage.VerifyPersistence() at Microsoft.Identity.Client.Extensions.Msal.MsalCacheHelper.VerifyPersistence() at Azure.Identity.MsalClientBase1.GetClientAsync(Boolean async, CancellationToken cancellationToken) at Azure.Identity.MsalClientBase1.GetClientAsync(Boolean async, CancellationToken cancellationToken) at Azure.Identity.MsalPublicClient.GetAccountsAsync(Boolean async, CancellationToken cancellationToken) at Azure.Identity.SharedTokenCacheCredential.GetAccountAsync(Boolean async, CancellationToken cancellationToken) at Azure.Identity.SharedTokenCacheCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)

[et1975]

I figured a workaround just now:

• have a Dockerfile just for running stuff locally (not a great start, but easier than the alternatives)
• that uses mcr.microsoft.com/azure-cli as the base image and
• in docker-compose has:

volumes:
 - ~/.azure:/root/.azure 

And in the code sets up the auth chain:

ChainedTokenCredential(ManagedIdentityCredential() or EnvironmentCredential(), AzureCliCredential())

This works, but would be great if we didn't need az cli in the first place.

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[jongio]

See here for how I do it, which is the same as you, but checkout the CLI install script in my dev container, it's a one liner.

RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash

VIDEO: https://youtu.be/oDNGs7B2g1A CODE: https://github.com/jongio/azureclicredentialcontainer

The only thing better than this would be local ManagedIdentity, but that isn't available right now. We have discussed it, but it opens issues that need to be fleshed out.

[nhart12]

Agreed, to be able use/mount IDE azure credentials when local testing would be awesome. Azure CLI bloats images by almost a gig

[jdthorpe]

See here for how I do it, which is the same as you, but checkout the CLI install script in my dev container, it's a one liner.

RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash

VIDEO: https://youtu.be/oDNGs7B2g1A CODE: https://github.com/jongio/azureclicredentialcontainer

The only thing better than this would be local ManagedIdentity, but that isn't available right now. We have discussed it, but it opens issues that need to be fleshed out.

@jongio, This worked for me up until I upgraded my Azure CLI to 2.33. Now it seems the windows host machine encrypts the tokens in a .bin file, but the linux azure CLI inside the container expects the unencrypted .json file, so I get a message inside the container stating Please run 'az login' from a command prompt to authenticate before using this credential. inside the container, but the same code running on the windows host fetches an access token without issue.

[philipwolfe]

@et1975 @jdthorpe @jongio @christothes I am running into this too. Here is what I came up with. We too need ways for a container running on a QA engineer machine to authenticate to Azure without checking credentials into SCC in a YAML file. Would love some feedback. My goal is to take the access token from the engineer and use it for this session...doesn't need to be long term like the EnvironmentCredential. https://github.com/philipwolfe/azure-sdk-for-net/commit/5dff08d75bdb41a1bef4618d2478adbfb6f02338 based on ideas from: https://stackoverflow.com/a/61498506/13122820

[jdthorpe]

@philipwolfe this solution may work for you for now. It essentially requires installing a previous version of the Azure CLI onto both the host machine and in the container, logging into Azure (az login) on the host machine, mapping the ~/.azrue directory into the container.

[goenning]

We're also using the CLI solution, but the az cli on developer machines is auto updating to the 2.33 version, so that means every day developers have to downgrade to 2.29.

Based on az cli docs, it's not meant to auto-upgrade by default, but apparently it is...

By default, auto-upgrade for Azure CLI is disabled. If you would like to keep up with the latest version, you can enable auto-upgrade through [configuration](https://docs.microsoft.com/en-us/cli/azure/config).

[tpanum]

Surreal to read that no progress has been made on such a fundamental problem for over a year. Much like the Python counter part (azure-identities), this package simply seems to be poorly designed, as it relies on some unversioned binary to function. Thus this binary dependency has to be baked in to the container images, despite serving no use in production.

The least destructive hack I have come up with is simply to retrieve secrets (e.g. access token) from my host machine (using Azure CLI) and pass it into my docker container using environment variables, and overrule the azure-identity clients, like so: docker run -e TOKEN=$(az account get-access-token --resource <resource-id> | jq -r .accessToken) my/fantastic-image.

[paupal1]

Also running into this issue... Is there a recommended workaround other than downgrading AzCli version?

[dmitriyse]

Was forced to write a tool that proxies the local tokens for local user (obtained from the DefaultAzureCredential) to the container through the same protocol as MSI are delivered to the ARC enabled servers. https://github.com/ClrCoder/ClrPro.AzureFX/releases/tag/v0.1.0

This tool should be executed from a developer account on port 40342

$env:ASPNETCORE_URLS="http://+:40342"
.\ClrPro.Azure.LocalCredentialBridge.exe

Then container should have the next env, volumes:

docker ....
      -e IDENTITY_ENDPOINT=http://host.docker.internal:40342/metadata/identity/oauth2/token
      -e IMDS_ENDPOINT=http://host.docker.internal:40342 
      -v %USERPROFILE%/.LocalCredentialBridgeTokens:/var/opt/azcmagent/tokens:ro
      ...

And the DefaultAzureCredential will work inside the container.

[dmitriyse]

Ideally such functionality should be inside Visual Studio out of the box.

1. Docker containers development is a first-class feature of the Visual Studio
2. Azure secret-less resource access is a first-class feature of the Azure SDK
3. Azure connectivity from Visual-Studio again is a first class feature

Why developers should do the IDE enhancement job for the first class features to make them works together ?

Lack of support of zero secrets connectivity is appearing here and there. For example here there was also a problem https://github.com/dotnet/efcore/issues/26491

Please increase the priority of this feature request. It's spanning a year already.

[esimkowitz]

Hi @jongio, any updates here? While we would like to get all our developers working in Docker containers to improve compatibility with our production environments, requiring a complicated login process versus just running in VS is too much of a burden.

[asanjabi]

@esimkowitz one workaround is to mount a volume that's shared between all containers, you'd have to connect to one and login once, but the rest will be fine after that. You would need to install the CLI on all the images, so there is that. Not ideal, but workable sample

[esimkowitz]

Frankly that seems like more work to explain to my devs and write troubleshooting docs for than to just tell them to test their changes separately against our Linux environments.

[stefanolsenn]

We fixed it by injecting the environment variables into the containers: in our docker-compose file and using InTune to set the environment variables on all developer pc's.

      [...]
        environment:
         - AZURE_TENANT_ID
         - AZURE_CLIENT_ID
         - AZURE_CLIENT_SECRET
      [...]

[esimkowitz]

That kind of fix won't work for us. We do not store client credentials on local dev boxes, we need to have RBAC set up to someone's own account for any dev resources. We are able to use DefaultAzureCredential in Visual Studio with no issue, ideally this should pipe automatically into Docker when running locally.

[dmitriyse]

https://github.com/Azure/azure-sdk-for-net/issues/19167#issuecomment-1127081646

Please try this approach. Works good enough in our team. Visual Studio Credential get passed into containers.

~ 1/2 Year, all good, we forgot about this problem.

[mykola-yakovliev]

Just to add another argument to this problem: for someone (like me), who is new to development of cloud solutions using Azure and wants to try things out, it is a little bit frustrating experience to get an exception after you generate the project from a template and just want it to run with zero-configuration needed. Of course, it is not really much critical in my case, but from my point of view, people would expect it to work locally out-of-box equally with or without Docker.

[esimkowitz]

@jongio @jsquire any updates here on prioritizing this ask? I think this is a very popular scenario given the increasing focus on Dockerizing development and debug environments.

[jsquire]

Hi @esimkowitz. I have no insight; my role for this issue was only initial triage. Any updates would need to be provided by the assigned engineers, @christothes and @schaabs).

[dmitriyse]

Updated the LocalCredentialBridge utility: https://github.com/ClrCoder/ClrPro.AzureFX/releases/tag/v0.1.1

[jongio]

@dmitriyse - Do you have more info on that?

[jonms90]

I am running into the same issue for local development with docker containers in Visual Studio 2022 that relies on Azure services. In production/test I use Managed Identities without any issue, but that is not an option locally. The code uses the chained DefaultAzureCredential to support multiple credential providers. To summarize;

• EnvironmentalCredential: This works fine for User accounts, but not when MFA is enabled (which should always be enabled). The other option here is to use a Service Principal and pass in the client credentials using a .env file that is not checked in to source control. This works, but it is a hassle to manage with a lot of management overhead when your development teams starts to grow.
• ManagedIdentityCredential: As mentioned: works great for test/prod, but not available for local development.
• SharedTokenCacheCredential: There is little to no documentation on how this is supposed to work with a container?
• VisualStudioCredential: This is what I would expect to be the default developer experience in 2022, but it does not seem to be integrated with docker container support in VisualStudio. Ideally, logging into VS should be enough to authenticate regardless of running in a container or not. Unfortunately this is not how it works.
• Using Azure CLI. Another option that works with some hacks including mounting azure folders onto the running container, but the largest downside is that we have to include the Azure CLI in our container images. This dramaticly bloats our images and really is not an option considering the amount of images we create.
• InteractiveBrowserCredential does not seem to do anything when running in a container context

Using Visual Studio 2022, Azure and Docker in combination should not be this complicated. This seems like a very basic setup that will hit everyone trying to containerize their cloud-native applications. I must be missing something obvious.

[jonms90]

I guess the lesser evil is to use a Service Principal for each user, but that really does not seem to be the correct way of solving this issue.

[NCarlsonMSFT]

Update on this: I am a dev on the Container Tools team in VS and we are actively working on solving this issue; but unfortunately, I can't give you an exact timeline for when support will ship. Until then I have two samples to try and make the current experience more bearable: EnvironmentCredentialExample and AzureCliCredentialExample. Both use a combination of PowerShell scripts and debugging customizations to make the process of authenticating in development containers as straight forward as possible.

[Augustukas]

one more workaround described here https://endjin.com/blog/2022/09/using-azcli-authentication-within-local-containers

[BC89]

Incredibly frustrating. MS pushing Dockerized approach in all the VS2002 marketing BS and something as fundamental as this breaks down.

[BC89]

Looks like 1.9.0-beta.2 just hit and this still hasn't been addressed.

[NCarlsonMSFT]

Update: Using the new Azure.Identity 1.9.0-beta.2 and Visual Studio 2022 17.6 Preview 1 the VisualStudioCredential should now work when using Visual Studio to Launch a .NET Core project in a Windows or Linux container.