AKS User Assigned Managed Identity - Unexpected content type

[hemantmehra14]

• Package Name: azure-core==1.18.0, azure-identity==1.6.1
• Operating System: Ubuntu
• Python Version: 3.7

Issue When using User assigned managed identity on Azure Kubernetes Cluster where Kubernetes job pulls an image from Azure Container Registry, I encountered the following error:

ClientAuthenticationError(ManagedIdentityCredential: Unexpected content type "text/plain; charset=utf-8")

Python Traceback:

INFO - azure.identity._credentials.managed_identity
ManagedIdentityCredential will use IMDS

WARNING - azure.identity._internal.get_token_mixin
ImdsCredential.get_token failed: Unexpected content type "text/plain; charset=utf-8"
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/azure/identity/_internal/get_token_mixin.py", line 72, in get_token
    token = self._request_token(*scopes)
  File "/usr/local/lib/python3.7/site-packages/azure/identity/_credentials/imds.py", line 93, in _request_token
    six.raise_from(ClientAuthenticationError(message=ex.message, response=ex.response), None)
  File "<string>", line 3, in raise_from
azure.core.exceptions.ClientAuthenticationError: Unexpected content type "text/plain; charset=utf-8"

WARNING - azure.identity._internal.decorators
ManagedIdentityCredential.get_token failed: Unexpected content type "text/plain; charset=utf-8"
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/azure/identity/_internal/decorators.py", line 27, in wrapper
    token = fn(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/azure/identity/_credentials/managed_identity.py", line 88, in get_token
    return self._credential.get_token(*scopes, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/azure/identity/_internal/get_token_mixin.py", line 72, in get_token
    token = self._request_token(*scopes)
  File "/usr/local/lib/python3.7/site-packages/azure/identity/_credentials/imds.py", line 93, in _request_token
    six.raise_from(ClientAuthenticationError(message=ex.message, response=ex.response), None)
  File "<string>", line 3, in raise_from
azure.core.exceptions.ClientAuthenticationError: Unexpected content type "text/plain; charset=utf-8"

Dockerfile for image in ACR

FROM prefecthq/prefect:latest-python3.7
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt

Code

from azure.identity import ManagedIdentityCredential
credential = ManagedIdentityCredential()

The above mentioned code runs in Kubernetes container which is created from the above dockerfile image.

Additional context Using az aks pod-identity add feature of AKS.

[xiangyan99]

Thanks for reaching out.

Can you enable debug logging and share us the log file?

[hemantmehra14]

Actually I am using prefect to run my python code on Kubernetes and I could only get INFO and WARNING logs out of it. Not sure how to get DEBUG logs too.

I used below code to get azure.identity logs:

import logging
az_logger = logging.getLogger('azure.identity')
az_logger.setLevel(logging.DEBUG)
log_stream = logging.StreamHandler(sys.stdout)
az_logger.addHandler(log_stream)

Following are INFO and WARNING logs.

INFO - azure.identity._credentials.managed_identity
ManagedIdentityCredential will use IMDS

WARNING - azure.identity._internal.get_token_mixin
ImdsCredential.get_token failed: Unexpected content type "text/plain; charset=utf-8"
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/azure/identity/_internal/get_token_mixin.py", line 72, in get_token
    token = self._request_token(*scopes)
  File "/usr/local/lib/python3.7/site-packages/azure/identity/_credentials/imds.py", line 93, in _request_token
    six.raise_from(ClientAuthenticationError(message=ex.message, response=ex.response), None)
  File "<string>", line 3, in raise_from
azure.core.exceptions.ClientAuthenticationError: Unexpected content type "text/plain; charset=utf-8"

WARNING - azure.identity._internal.decorators
ManagedIdentityCredential.get_token failed: Unexpected content type "text/plain; charset=utf-8"
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/azure/identity/_internal/decorators.py", line 27, in wrapper
    token = fn(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/azure/identity/_credentials/managed_identity.py", line 88, in get_token
    return self._credential.get_token(*scopes, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/azure/identity/_internal/get_token_mixin.py", line 72, in get_token
    token = self._request_token(*scopes)
  File "/usr/local/lib/python3.7/site-packages/azure/identity/_credentials/imds.py", line 93, in _request_token
    six.raise_from(ClientAuthenticationError(message=ex.message, response=ex.response), None)
  File "<string>", line 3, in raise_from
azure.core.exceptions.ClientAuthenticationError: Unexpected content type "text/plain; charset=utf-8"```

[xiangyan99]

You can add

import logging
logging.basicConfig(filename='c:/temp/log.log', level=logging.DEBUG)

And when you create the credential object, please add logging_enable=True.

Thank you.

[mattmshell]

This feels similar to #25450 which has been reproduced and escalated the the product team. In the end my service URI/scope needed to be lowercase.

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!