Tests for wait signal are flaky on MacOS

[nicoddemus]

Unfortunately requires access to a Mac to investigate/debug this further.

Marked the flaky tests as xfail. logs_98.zip

[bitlogik]

Thanks for your effort on this useful tool. We faced this exact issue when porting the guardata app to MacOSX. At first we were very disappointed and puzzled because the app looks to behaves right manually, but the pytest-qt were a bloodbath. The we discovered this issue, and without digging into the root cause, we decided to write our own basic wait_until method for MacOSX. The code comment "wait timeout then assert callback" was not properly updated as we improved the method and chopped the waiting time in 20 slices. See there : https://github.com/bitlogik/guardata/blob/35355237d35a2435f33f5f14ff24f047a7246244/tests/client/gui/conftest.py#L186

The point is there's one issue (that I don't fully understand). It requires a qtbot.wait() right before the wait_until, else the waiting times in this homemade wait_until are blocking and are barely used for rendering in the waiting "truth" function, and not even the drawing of the preceding thing (before the def "truth" function)

Something like (working in Linux and Windows) :

letsdoaction()
def _entry_available():
    assert rowCount() == 2
await aqtbot.wait_until(_entry_available)

needs to be with an additional wait to work (have the time to render), else the render is only done after the waituntil and it fails. It is exactly like our qtbot.wait inside our method are blocking.

letsdoaction()
def _entry_available():
    assert rowCount() == 2
await aqtbot.wait(200)
await aqtbot.wait_until(_entry_available)

The issue is not about the time required is longer in Mac, but our homemade wait_until blocks until the end the rendering. We're not "async" experts, so we may have missed some trivial point about blocking and parallel running.

See the diff there : https://github.com/bitlogik/guardata/commit/13ee338c15694263d3a6a46bb9468c0f3e7f103a

The MacOSX tests are drastically reduced, to a bare minimum, but we handled to manage some with this homemade wait_until method. We didn't make the changes required (adding aqtbot.wait) to all the tests but only to a minimal subset, as we expect that it could be fixed here in pytest-qt. Our goal was to have some minimal confidence in our Mac version of our app, having a very basic test set (on top of many manual tests), so we have released the first Mac version.

One way of seeing this issue, is maybe this is just an issue for no reason that qtbot.wait is sometimes blocking on Mac. But as we can read there, we also noticed very weird behavior with the timing. For example, over ~300ms, it can block for ever.

Regarding this issue and the fact that you don't have access to a Mac, we are considering renting a cloud mac for you for 1 or 2 months, so you (or the assigned expert) can take a look on what happens in details in a Mac. Let us know if you have time to process with this plan, and we'll make a Mac at your disposal.

[The-Compiler]

FWIW I do have access to a Mac and I do have issues with wait_signal blocking forever, or at least >10 seconds for a trivial signal invocation (at least on GitHub Actions CI).

Personally I do lack the time (and perhaps the experience, though not entirely sure about this one) to investigate further. :sweat_smile:

@bitlogik If you happen to be interested in a paid fix for this (as long as @nicoddemus is okay with it), please reach out to [email protected] - otherwise I still want to dig into it at some point, but I can't promise if/when I'll have time to take a closer look.

[nicoddemus]

... as long as @nicoddemus is okay with it...

Definitely ok with this, in fact no need to ever ask me permission for this @The-Compiler: you've contributed to this project as much as me. 👍

[bitlogik]

Thanks for your suggestion, we are not rushed to fix this issue. Our support here, on top of our technical feedback/insight, is only to provide a Mac remote VM in case some developers don't have Mac.

On the issue, we now feel that this is related to the extensive use of _exec() in our code base, where the GUI work "behind" the GUI is blocked, and on Mac the problem is aggravated, meaning the GUI has not the time to fully render or instantiate the modal, then the modal can freeze, display a modal without text, or disable input on its button. We started to address this issue about modal freezing (faced in manual testing, without pytest), by changing many _exec() to open() where we can, and adding the code below when we need (for now) a _exec() call

self.show_top()
QCoreApplication.processEvents()

See there the changes : https://github.com/bitlogik/guardata/commit/eb20ea9104d597d48d3eab16bc31454437a68a18 So the present issue might not be strictly in pytest-qt, and I even feel maybe not only on MacOS, this might be related to "blocking" modals which freeze the GUI as the backend painting and app running stop. It seems to stop earlier on Mac so it bring the issue, as remarked on this linked channel "QTrio [sic, QT I guess] has a bunch of warnings on macOS including comments about being slow", so one explanation of the root cause is that rendering or instantiation take a lot more time on Mac compared to Linux or Windows, and this brings this freezing issue on blocking modals. Because the app didn't have the time to fully paint or instantiate the modal and when it is finally halted with _exec, the app stops at some point and that freezes the modal, or render it incomplete, or sometimes registered callbacks for button are not registered.

So at the end, it may not related to timer management, but let's not close the door at this stage, maybe different timers units also play a role. Maybe the timers are just stretched because of different time scales and the freezing bug is an additional and separate issue. This can also comes from an underlying issue in QT on Mac where the app is halted too early when it process a blocking _exec modal.

For our side, we plan to fully fix this related issue of GUI behavior to turn all our modals to "non-blocking" ones, means changing exec_() to show() and adding callback functions linked on buttons, instead of waiting the return result of the _exec "blocking" modal. This is mostly required on modal where the user has a "yes/no" choice. We feel that this can even fix this current issue with pytest-qt.