azure-cli icon indicating copy to clipboard operation
azure-cli copied to clipboard

{testsdk} Initialize random config dir in `setUp`

Open jiasli opened this issue 9 months ago • 3 comments

Fix #28848

Description According to https://docs.python.org/3/library/unittest.html#unittest.TestCase.tearDown, tearDown is a counterpart of setUp, meaning anything that is done in tearDown should have its equivalent in setUp.

This PR solves 2 problems:

  1. config_dir leak: By injecting print(os.getpid(), method_name) into azure.cli.testsdk.base.ScenarioTest.__init__ and running azdev test in --series mode , we can see pytest first creates a unittest.TestCase with method_name="runTest", and runs no test with it:

    $ azdev test test_containerapp_manualjob_withsecret_crudoperations_e2e --series
    ...
    collecting ... 1098 runTest
    1098 test_containerapp_manualjob_withsecret_crudoperations_e2e
    collected 1 item
    
    src/azure-cli/azure/cli/command_modules/containerapp/tests/latest/test_containerappjob_with_secrets. py::ContainerAppJobsSecretsOperationsTest::test_containerapp_manualjob_withsecret_crudoperations_e2e PASSED [100%]
    

    As no test method is run for runTest, tearDown is not called, leading to random config dir not being deleted.

  2. patch.dict pollution: From the above log, we can see azure.cli.testsdk.base.ScenarioTest.__init__ is run at pytest's collecting staging, in the same process (1098), but tearDown is executed after test methods have been finished. Therefore, this patch.dict(os.environ, {'AZURE_CONFIG_DIR': config_dir}) statement pollutes other test classes' __init__ methods while pytest collects them.

Related:

  • https://github.com/Azure/azure-cli/pull/25689
  • https://github.com/Azure/azure-cli/pull/26475
  • https://github.com/Azure/azure-cli/pull/28673

jiasli avatar Apr 28 '24 21:04 jiasli

❌AzureCLI-FullTest
️✔️acr
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️acs
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️advisor
️✔️latest
️✔️3.11
️✔️3.9
️✔️ams
️✔️latest
️✔️3.11
️✔️3.9
️✔️apim
️✔️latest
️✔️3.11
️✔️3.9
️✔️appconfig
️✔️latest
️✔️3.11
️✔️3.9
️✔️appservice
️✔️latest
️✔️3.11
️✔️3.9
️✔️aro
️✔️latest
️✔️3.11
️✔️3.9
️✔️backup
️✔️latest
️✔️3.11
️✔️3.9
️✔️batch
️✔️latest
️✔️3.11
️✔️3.9
️✔️batchai
️✔️latest
️✔️3.11
️✔️3.9
️✔️billing
️✔️latest
️✔️3.11
️✔️3.9
️✔️botservice
️✔️latest
️✔️3.11
️✔️3.9
️✔️cdn
️✔️latest
️✔️3.11
️✔️3.9
️✔️cloud
️✔️latest
️✔️3.11
️✔️3.9
️✔️cognitiveservices
️✔️latest
️✔️3.11
️✔️3.9
️✔️compute_recommender
️✔️latest
️✔️3.11
️✔️3.9
️✔️config
️✔️latest
️✔️3.11
️✔️3.9
❌configure
❌latest
❌3.11
Type Test Case Error Message Line
Failed test_configure_global_defaults self = <azure.cli.command_modules.configure.tests.latest.test_configure.ConfigureGlobalDefaultsTest testMethod=test_configure_global_defaults>

    def test_configure_global_defaults(self):
        # setiing the az configure defaults
>       self.cmd('configure --defaults global=global1')

src/azure-cli/azure/cli/command_modules/configure/tests/latest/test_configure.py:41: 
                                        

self = <azure.cli.command_modules.configure.tests.latest.test_configure.ConfigureGlobalDefaultsTest testMethod=test_configure_global_defaults>
command = 'configure --defaults global=global1', checks = None
expect_failure = False

    def cmd(self, command, checks=None, expect_failure=False):
        command = self._apply_kwargs(command)
>       return execute(self.cli_ctx, command, expect_failure=expect_failure).assert_with_checks(checks)
E       AttributeError: 'ConfigureGlobalDefaultsTest' object has no attribute 'cli_ctx'

src/azure-cli-testsdk/azure/cli/testsdk/base.py:200: AttributeError
azure/cli/command_modules/configure/tests/latest/test_configure.py:38
❌3.9
Type Test Case Error Message Line
Failed test_configure_global_defaults self = <azure.cli.command_modules.configure.tests.latest.test_configure.ConfigureGlobalDefaultsTest testMethod=test_configure_global_defaults>

    def test_configure_global_defaults(self):
        # setiing the az configure defaults
>       self.cmd('configure --defaults global=global1')

src/azure-cli/azure/cli/command_modules/configure/tests/latest/test_configure.py:41: 
                                        

self = <azure.cli.command_modules.configure.tests.latest.test_configure.ConfigureGlobalDefaultsTest testMethod=test_configure_global_defaults>
command = 'configure --defaults global=global1', checks = None
expect_failure = False

    def cmd(self, command, checks=None, expect_failure=False):
        command = self._apply_kwargs(command)
>       return execute(self.cli_ctx, command, expect_failure=expect_failure).assert_with_checks(checks)
E       AttributeError: 'ConfigureGlobalDefaultsTest' object has no attribute 'cli_ctx'

src/azure-cli-testsdk/azure/cli/testsdk/base.py:200: AttributeError
azure/cli/command_modules/configure/tests/latest/test_configure.py:38
️✔️consumption
️✔️latest
️✔️3.11
️✔️3.9
️✔️container
️✔️latest
️✔️3.11
️✔️3.9
️✔️containerapp
️✔️latest
️✔️3.11
️✔️3.9
️✔️core
️✔️2018-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️cosmosdb
️✔️latest
️✔️3.11
️✔️3.9
️✔️databoxedge
️✔️2019-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️dla
️✔️latest
️✔️3.11
️✔️3.9
️✔️dls
️✔️latest
️✔️3.11
️✔️3.9
️✔️dms
️✔️latest
️✔️3.11
️✔️3.9
️✔️eventgrid
️✔️latest
️✔️3.11
️✔️3.9
️✔️eventhubs
️✔️latest
️✔️3.11
️✔️3.9
️✔️feedback
️✔️latest
️✔️3.11
️✔️3.9
️✔️find
️✔️latest
️✔️3.11
️✔️3.9
️✔️hdinsight
️✔️latest
️✔️3.11
️✔️3.9
️✔️identity
️✔️latest
️✔️3.11
️✔️3.9
️✔️iot
️✔️2019-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️keyvault
️✔️2018-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️kusto
️✔️latest
️✔️3.11
️✔️3.9
️✔️lab
️✔️latest
️✔️3.11
️✔️3.9
️✔️managedservices
️✔️latest
️✔️3.11
️✔️3.9
️✔️maps
️✔️latest
️✔️3.11
️✔️3.9
️✔️marketplaceordering
️✔️latest
️✔️3.11
️✔️3.9
️✔️monitor
️✔️latest
️✔️3.11
️✔️3.9
️✔️mysql
️✔️latest
️✔️3.11
️✔️3.9
️✔️netappfiles
️✔️latest
️✔️3.11
️✔️3.9
️✔️network
️✔️2018-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️policyinsights
️✔️latest
️✔️3.11
️✔️3.9
️✔️privatedns
️✔️latest
️✔️3.11
️✔️3.9
️✔️profile
️✔️latest
️✔️3.11
️✔️3.9
️✔️rdbms
️✔️latest
️✔️3.11
️✔️3.9
️✔️redis
️✔️latest
️✔️3.11
️✔️3.9
️✔️relay
️✔️latest
️✔️3.11
️✔️3.9
️✔️resource
️✔️2018-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️role
️✔️latest
️✔️3.11
️✔️3.9
️✔️search
️✔️latest
️✔️3.11
️✔️3.9
️✔️security
️✔️latest
️✔️3.11
️✔️3.9
️✔️servicebus
️✔️latest
️✔️3.11
️✔️3.9
️✔️serviceconnector
️✔️latest
️✔️3.11
️✔️3.9
️✔️servicefabric
️✔️latest
️✔️3.11
️✔️3.9
️✔️signalr
️✔️latest
️✔️3.11
️✔️3.9
️✔️sql
️✔️latest
️✔️3.11
️✔️3.9
️✔️sqlvm
️✔️latest
️✔️3.11
️✔️3.9
️✔️storage
️✔️2018-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️synapse
️✔️latest
️✔️3.11
️✔️3.9
🔄telemetry
️✔️2018-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
🔄latest
🔄3.11
🔄3.9
️✔️util
️✔️latest
️✔️3.11
️✔️3.9
️✔️vm
️✔️2018-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9

️✔️AzureCLI-BreakingChangeTest
️✔️Non Breaking Changes

testsdk

yonzhan avatar Apr 28 '24 21:04 yonzhan