Unit testing - assert fails or passes if called before or after testbot.pop_message()

[jkornblum]

I am...

• [] Reporting a bug
• [ ] Suggesting a new feature
• [ ] Requesting help with running my bot
• [x] Requesting help writing plugins
• [ ] Here about something else

I am running...

• Errbot version: 5.1.3
• OS version: Windows 7
• Python version: 3.6.3
• Using a virtual environment: yes (conda)

Issue description

The call_count assertions on mocked objects change their behavior based on if they are called before or after testbot.pop_message(). Does this have to do with the fact that these are generators and pop_message() calls the generator function to next yield?

Steps to reproduce

plugin code

from errbot import BotPlugin, arg_botcmd, botcmd
import plugin_deps as pdep

@arg_botcmd('--rev', type=str, default=None)
@arg_botcmd('part', type=str)
def edb_testlist(self, msg, part, rev):
        edb_part = pdep.pyedb.build_part(part)
        yield "Okay, let me get the production test list for {}.".format(part)
        test_list = edb_part.return_test_names()
        if len(test_list.index):
            latest_rev = get_latest_rev()
            yield "Here are the tests for the latest used test revision {}.".format(latest_rev)
        else:
            yield "I don't see any tests for {}.".format(part)

test code (FAIL)

pytest_plugins = ["errbot.backends.test"]
extra_plugin_dir = '.'
from unittest.mock import MagicMock, patch

@patch('plugin_deps.pyedb', autospec=True)
def test_edb_testlist_no_tests(pyedb_mock, testbot):
    mock_part = MagicMock()
    pyedb_mock.build_part.return_value = mock_part

    testbot.push_message('!edb testlist QPA1234')

    assert 1 == pyedb_mock.build_part.call_count
    assert 1 == mock_part.return_test_names.call_count
    assert "Okay, let me get the production test list for QPA1234." == testbot.pop_message()
    assert "I don't see any tests for QPA1234." == testbot.pop_message()

test code (PASS) - switch the assert order

pytest_plugins = ["errbot.backends.test"]
extra_plugin_dir = '.'
from unittest.mock import MagicMock, patch

@patch('plugin_deps.pyedb', autospec=True)
def test_edb_testlist_no_tests(pyedb_mock, testbot):
    mock_part = MagicMock()
    pyedb_mock.build_part.return_value = mock_part

    testbot.push_message('!edb testlist QPA1234')

    assert "Okay, let me get the production test list for QPA1234." == testbot.pop_message()
    assert "I don't see any tests for QPA1234." == testbot.pop_message()
    assert 1 == pyedb_mock.build_part.call_count
    assert 1 == mock_part.return_test_names.call_count

[gbin]

I think I understand what is going on: push_message just sends it to a queue and the "wait that the call happened" logic in pop_message. This is simply a race condition.

Now I don't really know where we can go from there... do we want a testbot.wait_empty_queue() method or something similar?