MSI authentication is failing even though token GCM generated for the identity seems to be valid.

[romahamu]

Version

2.4.1

Operating system

Linux

OS version or distribution

Ubuntu 20.04

Git hosting provider(s)

Azure DevOps

Other hosting provider

No response

(Azure DevOps only) What format is your remote URL?

https://dev.azure.com/{org}

Can you access the remote repository directly in the browser?

Yes, I can access the repository

Expected behavior

Be able to successfully clone the git repo.

Works - Git clone with bearer token from az login az login --identity --username identity-client-id az account get-access-token Use above token for bearer authentication and git clone works.

Works - Git clone with bearer token from GCM auth log git -c http.extraheader="AUTHORIZATION: bearer $password" clone https://dev.azure.com/msazure/one/REMOVED

Does not work - Configure GCM to use managed identity [credential "https://dev.azure.com"] helper = helper = /home/aadmin/.dotnet/tools/git-credential-manager useHttpPath = true credentialStore = gpg azreposManagedIdentity = id://REMOVED-CLIENT-ID azreposCredentialType = oauth traceSecrets = true traceMsAuth = true trace = /tmp/git.log fatal: Authentication failed for 'https://dev.azure.com/mszure/one/REMOVED

So that means GCM auth is actually generating correct token for the identity, but clone is failing with auth error later? I have attached the logs for the triage.

Actual behavior

fatal: Authentication failed for 'https://dev.azure.com/mszure/one/REMOVED

Logs

gcm-diagnose.log git.log

[mjcheetham]

Thanks for the logs! In the logs, did you notice if the token returned as password=<REMOVED> subsequently works if you use the value as a http.extraheader?

Could you also try repeating this and capture the network logs from Git?

GIT_TRACE=1 GIT_TRACE_CURL=1 git clone <URL>

You should get a lot of text from cURL including remote server responses. I'd be interested to see what headers are being sent and returned w.r.t. auth.

[andyliuliming]

meet the same issue, but I'm sure that the msi I used have the contributor permission of our repo

error: 10:48:01.497669 http.c:652 <= Recv data: />.. </di 10:48:01.497673 http.c:652 <= Recv data: v>..

.. <div cla 10:48:01.497677 http.c:652 <= Recv data: ss="title">401 - Uh-oh, you do not have access.

..
10:48:01.497680 http.c:652 <= Recv data:

The request requires authenticat 10:48:01.497684 http.c:652 <= Recv data: ion.

..

..
10:48:01.497686 http.c:652 <= Recv data:

4/17/2024 10:48:01 AM (UTC)

..
10:48:01.497691 http.c:652 <= Recv data:

8b657683-af2d-4df0-9753-0afdf0ce8f2f

..
10:48:01.497694 http.c:652 <= Recv data:

.. Sign out and l 10:48:01.497716 http.c:652 <= Recv data: ogin with different account.. .. < 10:48:01.497720 http.c:652 <= Recv data: /div>.. .. ..

..

[romahamu]

did you notice if the token returned as password=<REMOVED> subsequently works if you use the value as a http.extraheader? Works - Git clone with bearer token from GCM auth log

@mjcheetham, yes password from GCM log works with http header. I gave example of that in the previous message. git -c http.extraheader="AUTHORIZATION: bearer $password" clone https://dev.azure.com/msazure/one/REMOVED

I will add the extra tracing and post it here.

[romahamu]

Attached the log and some excerpt added below for ref -

I am not sure if the UAMI is being masked here with aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa but thats not the UAMI id I pass in my git config. I do see valid UAMI client Id in the log though only once.

Error clearly says that UAMI does not have access but the UAMI is added as a user to the ADO project as well as the repository.

Token generated by GCM in the log is definitely valid and I have been able to clone using that same token via extraheader. So something is really off with GCM + UAMI + Ubuntu 20.04

18:06:13.964498 http.c:831 <= Recv data: The user 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa' is n 18:06:13.964504 http.c:831 <= Recv data: ot authorized to access this resource. 18:06:13.299352 http.c:831 <= Recv data: d>..

..

.. <div class= 18:06:13.299358 http.c:831 <= Recv data: "header">..

.. .. <img ........ 18:06:13.299484 http.c:831 <= Recv data: Azure DevOps

..

.. Service Status 18:06:13.299495 http.c:831 <= Recv data: .. Support.. @AzureDevOps.. .. 18:06:13.299512 http.c:831 <= Recv data:

..

..

.. <img class="logo" src= ........ 18:06:13.370039 http.c:831 <= Recv data: AAgAH4/wFTs+5SstXLiAAAAABJRU5ErkJggg==" />..

18:06:13.370043 http.c:831 <= Recv data: ..

.. <div class 18:06:13.370047 http.c:831 <= Recv data: ="title">401 - Uh-oh, you do not have access.

..
18:06:13.370051 http.c:831 <= Recv data:

The request requires authenticatio 18:06:13.370055 http.c:831 <= Recv data: n.

..

..
18:06:13.370059 http.c:831 <= Recv data:

4/17/2024 6:06:13 PM (UTC)

.. < 18:06:13.370063 http.c:831 <= Recv data: div>25190aa8-22c3-4ff5-860c-cf5072c718e1

..
18:06:13.370067 http.c:831 <= Recv data:

.. Sign out and login 18:06:13.370083 http.c:831 <= Recv data: with different account..

.. 18:06:13.370087 http.c:831 <= Recv data: .. .. .. ..

[andyliuliming]

@mjcheetham I think this is because the "Authorization: Basic id://blabla: oauth token" is sent out.

and actually the ado rest api can only works for the raw: "Authorzation: Bearer oauth token"

so in theory, this just can not work??

[this can repro on ubuntu 22.04, and latest git too]

[andyliuliming]

ok, I just find one way to make this work. we can only use the raw id in the azreposManagedIdentity the id://blabla and resource://blabla won't work.

[mjcheetham]

@mjcheetham I think this is because the "Authorization: Basic id://blabla: oauth token" is sent out.

Aha! Yes, that is likely the issue. Azure DevOps isn't happy with the username value since it includes the :// characters and id:// scheme overall.

Using the plain [guid] format for the Managed Identity ID should be fine as you've discovered.