azure-cli-dev-tools icon indicating copy to clipboard operation
azure-cli-dev-tools copied to clipboard

azdev test should warn users if they have configured defaults

Open adewaleo opened this issue 5 years ago • 1 comments

Some commands default to values specified by the configured default in the user's environment (set via az configure). See --resource-group and --locations as examples of optionals which support configured defaults.

This can lead to inconsistencies where a test succeeds locally, but fails in the https://github.com/Azure/azure-cli repo's CI.

azdev test should warn users that they have configured defaults set which can lead to inconsistent playback behavior of the generated test recordings.

adewaleo avatar Nov 19 '19 23:11 adewaleo

For a concrete example that was debugged with @adewaleo's help, one example where this can cause local tests to pass but CI builds to fail is when --location is configured.

If the tests do not have the --location value hardcoded into their test, the test picks up the dev's local Azure CLI configuration. Thus the test harness doesn't perform the HTTP request for available locations and the new recording has the HTTP request removed.

On the CI machine it does not have the same default configured so it will perform this request, which leads to the actual behavior to not match with the recordings.

Example: Azure-cli#10584

stevengum avatar Nov 19 '19 23:11 stevengum