medperf icon indicating copy to clipboard operation
medperf copied to clipboard

[BUG] unit tests fails if the active profile is not `default`

Open hasan7n opened this issue 2 years ago • 2 comments

Describe the bug when the local profile is set to test as the active one, unittests of test_utils.py fail when running all tests in one pytest session. If the tests are run individually, the tests will pass. This can be misleading for developers and time-consuming. Besides, it seems that something is changing globally during tests, which is incorrect.

To Reproduce Steps to reproduce the behavior:

  1. activate the test profile
  2. run pytest

Additional context

The problem is still not 100% clear to me, however, a couple of clues:

  1. Deleting test_profile.py resolves the issue.
  2. The problem starts to happen after merging this PR: #342
  3. This line from medperf.commands.profile import app in test_profile.py seems to be the issue. When commenting all the code in this file and keeping this import statement, the problem persists.

Possible explanation: After merging the mentioned PR, the decorator configurable was edited to modify config. When importing app in test_profile.py, the decorator seems to be executed. However, it is still not clear why test_utils.py is failing.

Solution tried but it didn't work: adding this fixture in conftest.py

@pytest.fixture(autouse=True)
def disable_set_custom_config(mocker):
    mocker.patch("medperf.utils.set_custom_config")

Note Good to keep this in mind if we encountered any weird unit-tests failures.

hasan7n avatar Jan 16 '23 18:01 hasan7n

could have been fixed by https://github.com/mlcommons/medperf/pull/516. confirmation required

hasan7n avatar Feb 26 '24 10:02 hasan7n

At least for me it works well since that PR, so confirm it is fixed:)

VukW avatar Feb 26 '24 15:02 VukW