azure-cli [Cloud] Perform cloud discovery with API Version 2022-09-01 and support `dataplaneEndpoints` for Azure Stack

Related command az cloud

Description Close https://github.com/Azure/azure-cli/issues/25815

Testing Guide

Populate dataplaneEndpoints from metadata:

az cloud register -n mycloud --endpoint-resource-manager https://management.azure.com/

Populate dataplaneEndpoints from local cloud config JSON file:

az cloud register -n mycloud --cloud-config @D:\cloud\cloud-config.json

Mar 16 '23 04:03 jiasli

️✔️acr

️✔️acs

️✔️advisor

️✔️ams

️✔️apim

️✔️appconfig

️✔️appservice

️✔️aro

️✔️backup

️✔️batch

️✔️batchai

️✔️billing

️✔️botservice

️✔️cdn

❌cloud

Type	Test Case	Error Message	Line
Failed	test_cloud_scenario	E vcr.errors.CannotOverwriteExistingCassetteException: Can't overwrite existing cassette ('/mnt/vss/_work/1/s/src/azure-cli/azure/cli/command_modules/cloud/tests/latest/recordings/test_cloud_scenario.yaml') in your current record mode ('once'). No match for the request (<Request (GET) https://management.azure.com//metadata/endpoints?api-version=2022-09-01>) was found. Found 1 similar requests with 1 different matcher(s) : 1 - (<Request (GET) https://management.azure.com/metadata/endpoints?api-version=2015-01-01>). Matchers succeeded : ['method', 'scheme', 'host', 'port', 'path'] Matchers failed : _custom_request_query_matcher - assertion failure : None During handling of the above exception, another exception occurred: E AssertionError: Can't overwrite existing cassette ('/mnt/vss/_work/1/s/src/azure-cli/azure/cli/command_modules/cloud/tests/latest/recordings/test_cloud_scenario.yaml') in your current record mode ('once'). No match for the request (<Request (GET) https://management.azure.com//metadata/endpoints?api-version=2022-09-01>) was found. Found 1 similar requests with 1 different matcher(s) : 1 - (<Request (GET) https://management.azure.com/metadata/endpoints?api-version=2015-01-01>). Matchers succeeded : ['method', 'scheme', 'host', 'port', 'path'] Matchers failed : _custom_request_query_matcher - assertion failure : None	azure/cli/command_modules/cloud/tests/latest/test_cloud.py:79

Type	Test Case	Error Message	Line
Failed	test_cloud_scenario	E vcr.errors.CannotOverwriteExistingCassetteException: Can't overwrite existing cassette ('/mnt/vss/_work/1/s/src/azure-cli/azure/cli/command_modules/cloud/tests/latest/recordings/test_cloud_scenario.yaml') in your current record mode ('once'). No match for the request (<Request (GET) https://management.azure.com//metadata/endpoints?api-version=2022-09-01>) was found. Found 1 similar requests with 1 different matcher(s) : 1 - (<Request (GET) https://management.azure.com/metadata/endpoints?api-version=2015-01-01>). Matchers succeeded : ['method', 'scheme', 'host', 'port', 'path'] Matchers failed : _custom_request_query_matcher - assertion failure : None During handling of the above exception, another exception occurred: E AssertionError: Can't overwrite existing cassette ('/mnt/vss/_work/1/s/src/azure-cli/azure/cli/command_modules/cloud/tests/latest/recordings/test_cloud_scenario.yaml') in your current record mode ('once'). No match for the request (<Request (GET) https://management.azure.com//metadata/endpoints?api-version=2022-09-01>) was found. Found 1 similar requests with 1 different matcher(s) : 1 - (<Request (GET) https://management.azure.com/metadata/endpoints?api-version=2015-01-01>). Matchers succeeded : ['method', 'scheme', 'host', 'port', 'path'] Matchers failed : _custom_request_query_matcher - assertion failure : None	azure/cli/command_modules/cloud/tests/latest/test_cloud.py:79

️✔️cognitiveservices

️✔️config

️✔️configure

️✔️consumption

️✔️container

️✔️core

️✔️cosmosdb

️✔️databoxedge

️✔️dla

️✔️dls

️✔️dms

️✔️eventgrid

️✔️eventhubs

️✔️feedback

️✔️find

️✔️hdinsight

️✔️identity

️✔️iot

️✔️keyvault

️✔️kusto

️✔️lab

️✔️managedservices

️✔️maps

️✔️marketplaceordering

️✔️monitor

️✔️netappfiles

️✔️network

️✔️policyinsights

️✔️privatedns

️✔️profile

️✔️rdbms

️✔️redis

️✔️relay

️✔️resource

️✔️role

️✔️search

️✔️security

️✔️servicebus

️✔️serviceconnector

️✔️servicefabric

️✔️signalr

️✔️sql

️✔️sqlvm

️✔️storage

️✔️synapse

️✔️telemetry

️✔️util

️✔️vm

Mar 16 '23 04:03 azure-client-tools-bot-prd[bot]

Cloud enhancement

Mar 16 '23 04:03 yonzhan

Although the main post mentioned that a new nested group called dataplaneEndpoints is being supported, there should also be support for new suffixes and endpoints that is not passed by a cloudconfig.json file during az cloud register. Currently, Azure CLI is using a deprecated method to populate the hard-coded metadata which should be updated to read all endpoints and suffixes: https://github.com/Azure/azure-cli/blob/091c6b1c40c19d4076376f9dfa0d4994b291a7c6/src/azure-cli/azure/cli/command_modules/cloud/custom.py#L69 The method is from this deprecated library: https://github.com/Azure/msrestazure-for-python/blob/8e3def8c9927a349596e575de93df76de7a3a2b6/msrestazure/azure_cloud.py#L204

Mar 30 '23 05:03 TheOnlyWei

If these new suffixes and endpoints are not passed by a cloudconfig.json file, how will Azure CLI even know them?

We can certainly update _populate_from_metadata_endpoint and make no longer deprecated. That's why this PR copies this function from msrestazure to Azure CLI.

Reading all endpoints and suffixes is simply against our modeling principal which enforces all endpoints to be an attribute of the cloud object. Of course, we can leverage setattr to programmatically change the attributes of the cloud object, this defeats the purpose of modeling.

Mar 30 '23 06:03 jiasli

@jiasli

The new endpoints and suffixes will appear in the GET request to the metadata endpoint URL.

Is there a reason there is an exception made for reading all endpoints and suffixes from the cloudconfig.json but not from the metadata endpoint? For example, the following code just adds all endpoints from cloudconfig.json: https://github.com/Azure/azure-cli/blob/091c6b1c40c19d4076376f9dfa0d4994b291a7c6/src/azure-cli/azure/cli/command_modules/cloud/custom.py#L46

Mar 30 '23 09:03 TheOnlyWei

I didn't write that code, but I guess it is because cloudconfig.json is designed/used specifically for Azure CLI. On the other hand, metadata endpoint is merely a supplemental way to extract information we want.

Mar 30 '23 10:03 jiasli

@jiasli Bumping this.

FYI 2022-09-01 is a breaking change for the metadata endpoint. Prior versions returned a JSON array of cloud endpoints for multiple environments. This version returns a single JSON object for the endpoints associated with the ARM instance being queried.

Is the goal to support both schemas with this for backwards compatibility?

Jul 12 '23 19:07 toefraz

azure-cli azure-cli copied to clipboard

[Cloud] Perform cloud discovery with API Version 2022-09-01 and support `dataplaneEndpoints` for Azure Stack

azure-cli
azure-cli copied to clipboard