First Behat scenario always fails on Github actions when testing with IOMAD

Open abias opened this issue 4 years ago • 1 comments

I am currently writing simple Behat tests for a plugin for IOMAD (the multi-tenancy Moodle distribution which can be found on https://github.com/iomad/iomad/).

Doing this, I encountered the issue that the first Behat scenario always fails on Github actions reproducibly.

The Github action run log tells me this:

Started at 02-02-2022, 14:43
-----------.......................................................

--- Failed hooks:

    BeforeStep # behat_hooks::before_step()
      The base URL (http://localhost:8000) is not a behat test site. Ensure that you started the built-in web server in the correct directory, or that your web server is correctly set up and started.
      
      +--[ HTTP/1.1 200 | http://localhost:8000/login/index.php | GoutteDriver ]
      |
      |  <!DOCTYPE html>
      |  
      |  <html  dir="ltr" lang="en" xml:lang="en">
      |  <head>
      |      <title>Acceptance test site: Log in to the site</title>
      |      <link rel="shortcut icon" href="http://localhost:8000/theme/image.php/iomadboost/theme/1643784081/favicon" />
      |      <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
      |  <meta name="keywords" content="moodle, Acceptance test site: Log in to the site" />
      |  <link rel="stylesheet" type="text/css" href="http://localhost:8000/theme/yui_combo.php?rollup/3.17.2/yui-moodlesimple.css" /><script id="firstthemesheet" type="text/css">/** Required in order to fix style inclusion problems in IE with YUI **/</script><link rel="stylesheet" type="text/css" href="http://localhost:8000/theme/styles.php/iomadboost/1643784081_1/all" />
      |  <script>
      |  //<![CDATA[
      |  var M = {}; M.yui = {};
      |  M.pageloadstarttime = new Date();
      |  M.cfg = {"wwwroot":"http:\/\/localhost:8000","sesskey":"vPifMJkfpC","sessiontimeout":"28800","themerev":"1643784081","slasharguments":1,"theme":"iomadboo...
      |

4 scenarios (3 passed, 1 failed)
66 steps (55 passed, 11 skipped)
0m16.85s (77.16Mb)

See https://github.com/lernlink/iomad-tool_companydomain/runs/5032827581?check_suite_focus=true#step:17:136

I already had a look at the code where this error message is thrown (https://github.com/iomad/iomad/blob/IOMAD_39_STABLE/lib/tests/behat/behat_hooks.php#L521), but I have no idea why it is thrown in this particular case.

Interestingly, this failure does not seem to be related to the particular scenario. If I switch the order of the scenarios in the feature file, it's still the first scenario in the first feature file which fails. I have now added a sacrificial dummy scenario on https://github.com/lernlink/iomad-tool_companydomain/blob/master/tests/behat/tool_companydomain_event.feature#L20 which will be the one which fails.

Interestingly, this failure only happens when testing with the IOMAD codebase (https://github.com/lernlink/iomad-tool_companydomain/blob/master/.github/workflows/moodle-plugin-ci.yml#L60). If I switch the codebase to Moodle core, the first scenario starts well and doesn't fail until it should test some IOMAD-specific step.

You may say now: "This isn't a problem of moodle-plugin-ci, it's a problem of IOMAD" and you may be right. However, I still hope for your insights why this failure message is thrown and what could go wrong in moodle-plugin-ci so that I will be able to investigate this further.

Thanks, Alex

Feb 02 '22 06:02 abias

Here's where the error message comes from: https://github.com/iomad/iomad/blob/a351daeb519fe9a4508e7cdbede6f9330146e9f8/lib/tests/behat/behat_hooks.php#L520

Here's the definition of that function: https://github.com/iomad/iomad/blob/a351daeb519fe9a4508e7cdbede6f9330146e9f8/lib/tests/behat/behat_hooks.php#L826-L828

Here's the place that changes the value from false to true: https://github.com/iomad/iomad/blob/a351daeb519fe9a4508e7cdbede6f9330146e9f8/lib/tests/behat/behat_hooks.php#L878-L880

My guess is that there are changes in IOMAD that alter the sequence of how the steps run, so you may want to check the last function to see if there is a hook calling that method. The documentation seems to indicate that "This must be the last BeforeStep hook in the setup.", so I'm guessing that the last BeforeStep is never running before the first test.

Feb 02 '22 23:02 jrchamp