Upgrading to pytest-django 3.9 breaks test

[ekallevik]

trafficstars

When upgrading to pytest-django 3.9.0 one of my tests breaks.

@pytest.mark.django_db
@patch("ntnui.utils.gsuite.GSuite.unsubscribe_from_singular_mailing_list")
@patch("ntnui.utils.gsuite.GSuite.unsubscribe_from_board_mailing_list")
@patch("ntnui.utils.gsuite.GSuite.unsubscribe_from_group_mailing_list")
@pytest.mark.parametrize(
    ("membership_type", "leave_group_list", "leave_board_list", "leave_singular_list"),
    [
        (MembershipType.member, True, False, False),
        (MembershipType.board_member, True, True, False),
        (MembershipType.leader, True, True, True),
    ],
)
def test_unsubscribe_from_mailing_lists_member(
    mock_unsubscribe_from_group_mailing_list,
    mock_unsubscribe_from_board_mailing_list,
    mock_unsubscribe_from_singular_mailing_list,
    membership_type,
    leave_group_list,
    leave_board_list,
    leave_singular_list,
    user,
):
    """Test that members are unsubscribed from appropriate mailing lists when leaving a group."""
    membership = MembershipFactory(type=membership_type, member=user)
    membership.delete_membership()

    assert mock_unsubscribe_from_group_mailing_list.called == leave_group_list
    assert mock_unsubscribe_from_board_mailing_list.called == leave_board_list
    assert mock_unsubscribe_from_singular_mailing_list.called == leave_singular_list

As you can see above, the test utilizes both mocking and parametrization. It seems like pytest-django 3.9 has some issues with the mocks. Furthermore, the ordering of the tests impacts the results. When running the test suit, this tests fails. When running the test suite once more, with the --failed-first option the test does not break.

[ekallevik]

When inspecting the code run pdb I have found that Python actually enters the methods I am trying to mock, that is ntnui.utils.gsuite.GSuite.unsubscribe_from_singular_mailing_list is called when mock_unsubscribe_from_singular_mailing_list should be used. This behaviour differs from pytest-django 3.8, where mock_unsubscribe_from_singular_mailing_list is used.

[blueyed]

@ekallevik Make sure that it is not the pytest version that triggers this, and then git-bisect it (pytest-django or pytest). Also a minimal reproducible example would be useful, of course - preferably in the form of a test for pytest-django.

[ekallevik]

I have pinned the pytest version to 5.4.3, but the bug still persists when upgrading pytest-django. You want me to clone this repo and running git bisect on it to find the commit that introduces the problem? How can I use a local build of pytest-django when running pytest on my own codebase?

[blueyed]

You can install it as an editable: pip install -e /path/to/pytest-django, then use git bisect … on the repo.