azure-cli icon indicating copy to clipboard operation
azure-cli copied to clipboard
verified publisher icon Azure

az login --identity fails on an AzureML compute instance: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Open leosannlsn opened this issue 3 years ago • 10 comments

Describe the bug

Command Name az login --identity

Errors:

The command failed with an unexpected error. Here is the traceback:
Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/opt/az/lib/python3.10/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/profile/custom.py", line 122, in login
    return profile.login_with_managed_identity(username, allow_no_subscriptions)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/_profile.py", line 234, in login_with_managed_identity
    msi_creds = MSIAuthenticationWrapper(resource=resource)
  File "/opt/az/lib/python3.10/site-packages/msrestazure/azure_active_directory.py", line 592, in __init__
    self.set_token()
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/auth/adal_authentication.py", line 75, in set_token
    super().set_token()
  File "/opt/az/lib/python3.10/site-packages/msrestazure/azure_active_directory.py", line 596, in set_token
    self.scheme, _, self.token = get_msi_token_webapp(self.resource, self.msi_conf)
  File "/opt/az/lib/python3.10/site-packages/msrestazure/azure_active_directory.py", line 548, in get_msi_token_webapp
    token_entry = result.json()
  File "/opt/az/lib/python3.10/site-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "/opt/az/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/opt/az/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/az/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

  • Create a compute instance in AzureML (mine is STANDARD_DS11_V2)
  • Start the compute instance and click on the Terminal link to start a session
  • Run az login --identity as suggested by the welcome message

Expected Behavior

az login --identity should log me in without failures

Environment Summary

Linux-5.15.0-1017-azure-x86_64-with-glibc2.31, Ubuntu 20.04.4 LTS
Python 3.10.5 ("base" conda environment)
Installer: DEB

azure-cli 2.39.0 *

Extensions:
ml 2.11.0

Dependencies:
msal 1.18.0b1
azure-mgmt-resource 21.1.0b1

Additional Context

leosannlsn avatar Nov 17 '22 08:11 leosannlsn

@jiasli for awareness

yonzhan avatar Nov 17 '22 09:11 yonzhan

This is still an issue for me and the solution to #20853 did not help in my case. This is my stack trace after using az login -i on an azure ml compute instance

@bebound @yonzhan @jiasli

The command failed with an unexpected error. Here is the traceback:
Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/anaconda/envs/azureml_py38/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/anaconda/envs/azureml_py38/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/anaconda/envs/azureml_py38/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/cli/command_modules/profile/custom.py", line 122, in login
    return profile.login_with_managed_identity(username, allow_no_subscriptions)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/cli/core/_profile.py", line 234, in login_with_managed_identity
    msi_creds = MSIAuthenticationWrapper(resource=resource)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/msrestazure/azure_active_directory.py", line 592, in __init__
    self.set_token()
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/cli/core/auth/adal_authentication.py", line 75, in set_token
    super().set_token()
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/msrestazure/azure_active_directory.py", line 596, in set_token
    self.scheme, _, self.token = get_msi_token_webapp(self.resource, self.msi_conf)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/msrestazure/azure_active_directory.py", line 548, in get_msi_token_webapp
    token_entry = result.json()
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

julyvee avatar Nov 21 '22 11:11 julyvee

@julyvee your issue is similar to #23978

bebound avatar Nov 22 '22 02:11 bebound

@bebound Yes, but there is no real solution in that ticket. What does "Taking extra steps to login" even mean.

julyvee avatar Nov 24 '22 10:11 julyvee

@julyvee jiasli send you an email, could you please take a look?

bebound avatar Nov 25 '22 02:11 bebound

@bebound Could you please post the solution here? Having the same issue and have no clue what does "Taking extra steps to login" mean as well.

trojan03 avatar Nov 29 '22 14:11 trojan03

I hit this too just now. Exact same issue. Any steps to fix?

csm10495 avatar Jan 14 '23 00:01 csm10495

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github.

Issue Details

Describe the bug

Command Name az login --identity

Errors:

The command failed with an unexpected error. Here is the traceback:
Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/opt/az/lib/python3.10/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/profile/custom.py", line 122, in login
    return profile.login_with_managed_identity(username, allow_no_subscriptions)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/_profile.py", line 234, in login_with_managed_identity
    msi_creds = MSIAuthenticationWrapper(resource=resource)
  File "/opt/az/lib/python3.10/site-packages/msrestazure/azure_active_directory.py", line 592, in __init__
    self.set_token()
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/auth/adal_authentication.py", line 75, in set_token
    super().set_token()
  File "/opt/az/lib/python3.10/site-packages/msrestazure/azure_active_directory.py", line 596, in set_token
    self.scheme, _, self.token = get_msi_token_webapp(self.resource, self.msi_conf)
  File "/opt/az/lib/python3.10/site-packages/msrestazure/azure_active_directory.py", line 548, in get_msi_token_webapp
    token_entry = result.json()
  File "/opt/az/lib/python3.10/site-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "/opt/az/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/opt/az/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/az/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

  • Create a compute instance in AzureML (mine is STANDARD_DS11_V2)
  • Start the compute instance and click on the Terminal link to start a session
  • Run az login --identity as suggested by the welcome message

Expected Behavior

az login --identity should log me in without failures

Environment Summary

Linux-5.15.0-1017-azure-x86_64-with-glibc2.31, Ubuntu 20.04.4 LTS
Python 3.10.5 ("base" conda environment)
Installer: DEB

azure-cli 2.39.0 *

Extensions:
ml 2.11.0

Dependencies:
msal 1.18.0b1
azure-mgmt-resource 21.1.0b1

Additional Context

Author: leosannlsn
Assignees: -
Labels:

Account, Service Attention, Machine Learning, customer-reported, feature-request, Auto-Assign, Azure CLI Team
Milestone: Backlog

ghost avatar Jan 16 '23 03:01 ghost

Routing to ML team to provide further guidance on what "Taking extra steps to login" is.

jiasli avatar Jan 16 '23 03:01 jiasli

Hi, we are also experiencing the same issue with the compute instances for most of the users in our AzureML workspaces. Interestingly for compute instances that are assigned to some other users (including my account), we are not able to reproduce this issue. @jiasli @azureml-github can you please share guidance on this? Happy to share further details to troubleshoot.

tunayokumus avatar Jan 17 '23 17:01 tunayokumus

Are we mean to assign a managed identity to compute instances? the notes on starting a terminal instance in Azure ML Studio are

Welcome to the Azure Machine Learning terminal

Enter "git clone [url]" to clone a repo Enter "git --help" to learn about Git CLI. To learn more about integrating Git with the Azure Machine Learning terminal, navigate here: https://learn.microsoft.com/en-us/azure/machine-learning/concept-train-model-git-integration#clone-git-repositories-into-your-workspace-file-system Enter "az ml --help" to learn about Azure ML CLI v2

Note: Use "az login --identity" instead of "az login" to avoid device code authentication

However there is no mention of the need for a managed identity.

jwgwalton avatar Feb 16 '23 10:02 jwgwalton

Faced the same issue described here with az login --identity, even with creating the compute instance with a system-assigned identity. I managed to login using the identity by addind the --username flag and passing the managed identity's application ID, like this: az login --identity --username <application-id-from-aad-enterpriseapplication-for-the-managed-identity>

Question is - why does the ID need to be specified if it's a system-assigned managed identity tied to the compute instance?

NSimpragaVolur avatar Mar 28 '23 11:03 NSimpragaVolur

@jiasli I have the same issue but when running cat ~/.azure/msal_token_cache.json (other thread, similar bug 20853) the file is not empty. The login fails when doing az login --identity

natasha-savic-msft avatar Apr 05 '23 05:04 natasha-savic-msft

What solved the problem for me was to restart the compute instance and log in via az login.

natasha-savic-msft avatar Apr 06 '23 01:04 natasha-savic-msft

What solved the problem for me was to restart the compute instance and log in via az login.

Does az login log you in with the system-assigned identity or with your own personal account?

NSimpragaVolur avatar Apr 10 '23 08:04 NSimpragaVolur

Hi @julyvee, Did you solve this issue?

Dave-Bouchard avatar Jun 12 '23 14:06 Dave-Bouchard

We just faced the same issue. It seems that settings for identity-based auth had not been set up correctly, that's one thing. But also the fact that the compute instance had been created by another user caused problems.

There are two things that it took to resolve:

  1. Copy settings for identity-based auth to environment variables: For that run while read -r line ; do export $line; done < /etc/environment.sso in your shell. This will copy settings for authentication endpoints.

  2. The compute instance in question had been created by someone else for me. In AzureML Studio, go to the compute instance and click on "Jupyter" in the "Applications" column. This will bring up a warning message, asking you to click if you trust this compute instance. Click to confirm that.

Once both those steps had been done, I was able to run az login --identity correctly.

So in way I guess the error message that I received was masking that "do you trust this compute instance" problem.

ant0nsc avatar Jun 29 '23 15:06 ant0nsc

Experienced this issue again on a brand new compute.

Then tried suggestion from this comment and step 2 didn't do anything (I didn't see the warning message) but step 1, saying run while read -r line ; do export $line; done < /etc/environment.sso in terminal, made az loging --identity work (not throw an error). However, whenever a new terminal is created this fix needs to be applied again.

I did notice that the az login -i said that i am now logged in with the identity of the virtual machine - is that expected/desired? I thought that all things in Azure ML are done via the user's identity. Note that for my use cases of azure.identity i need to loging into az as a user, not at the VM's identity.

@yonzhan, @jiasli, @bebound - i guess you're from the relevant team - could you please provide guidance on how things should be done? Also, could you propagate this information to the relevant Azure ML folks so that the need for this workaround can be removed?

astafan8 avatar Aug 01 '23 10:08 astafan8