buzztrax icon indicating copy to clipboard operation
buzztrax copied to clipboard

Make UI tests work on github actions

Open ensonic opened this issue 1 year ago • 2 comments

We still can't run the UI tests and this makes the whole CI run super slow (as each test runs into a timeout).

Locally the tests mostly work:

  • debian testing: 84%: Checks: 113, Failures: 0, Errors: 18
  • opensuse tumbleweed: ...

The test setup believes it has successfully setup the xvfb:

0:24:22.152876977 72611 0x55f86d997c70 INFO                bt-check tests/bt-check-ui.c:221:check_setup_test_display: display 0x55f86d9e85e0,":0" is active

It is not even clear if the xvfb setup is to blame. A lot of the tests have these as the last log lines:

0:26:02.382873958 72703 0x55f86d997c70 INFO                 bt-edit src/ui/edit/main-page-sequence.c:1413:sequence_table_init: preparing sequence table
0:26:02.383991664 72703 0x55f86d997c70 INFO                 bt-core src/lib/core/settings.c:408:bt_settings_make: return cached settings object 0x55f86dd10560,ref_ct=3,floating=0
0:26:02.384494252 72703 0x55f86d997c70 INFO      GST_PLUGIN_LOADING ../gst/gstplugin.c:987:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so" loaded
0:26:02.384515702 72703 0x55f86d997c70 INFO     GST_ELEMENT_FACTORY ../gst/gstelementfactory.c:489:gst_element_factory_create_with_properties: creating element "playbin"

The test itself seem to actually work and finish quickly (we see cleanup markers in the log). It feels more like the something in the test hardness hangs (e.g. taking down the xvfb server)

Completely alternative idea:

We could try to use a matrix to run groups of tests in parallel. https://stackoverflow.com/questions/59175332/using-output-from-a-previous-job-in-a-new-one-in-a-github-action https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs

ensonic avatar Mar 02 '24 13:03 ensonic

With the skipped UI test, CI run finishes in ~6m (instead of 45m !):

https://github.com/Buzztrax/buzztrax/actions/runs/8123130041

ensonic avatar Mar 02 '24 14:03 ensonic

One of the test that also fail locally on debian is:

BT_CHECKS="test_bt_edit_application_new_song" make bt_edit.check

When I run it with BT_CHECK_NO_XVFB=1 no window appears. Maybe this runs too quickly to even get any event loop to kick in?

ensonic avatar Mar 09 '24 19:03 ensonic