cli
cli copied to clipboard
Use Go for integration tests; test output exactly
Tested on the GitLab side
Rationale:
- Go seems to be fine at running these kinds of tests, and it removes the need to worry about Python and Selenium, which have not been working on GitLab, and which represent unnecessary barriers to testing on other platforms.
- Since we can replace the CLI's configuration, we should be able to predict / test the output of some commands.
Awkwardnesses were:
- I modified the root command to accept the
versionfrom the YAML configuration file - and that overrides the version from GoReleaser. - I've set
cd /in the generate_test_data.sh script, so that the CLI doesn't auto-detect the "current project" and use that to disable various commands, such asxdebug(disabled for non-PHP projects) anddrush(for disabled for non-Drupal projects). - Locally my machine keeps wanting to remove the latest feature, which is the
--quiet(-q) option, from the test help output. I don't understand this.
Follow up steps could be:
- Build or download PHP and the Legacy CLI on GitHub Actions, so that the integration tests can be run there.
- Test authenticated commands, complications are:
- It's hard to test against a known existing project. But it takes a long time (and is expensive) to create a new project.
- Output becomes less predictable: we could use a template, or just test for specific output with
assert.Contains()orassert.Regexp(). - Performance, and flakiness in general: perhaps these tests should be
allow_failureso that they do not block work.
- Test on multiple operating systems, such as Linux, Windows and MacOS.