cli icon indicating copy to clipboard operation
cli copied to clipboard

Use Go for integration tests; test output exactly

Open pjcdawkins opened this issue 1 year ago • 0 comments

Tested on the GitLab side

Rationale:

  • Go seems to be fine at running these kinds of tests, and it removes the need to worry about Python and Selenium, which have not been working on GitLab, and which represent unnecessary barriers to testing on other platforms.
  • Since we can replace the CLI's configuration, we should be able to predict / test the output of some commands.

Awkwardnesses were:

  • I modified the root command to accept the version from the YAML configuration file - and that overrides the version from GoReleaser.
  • I've set cd / in the generate_test_data.sh script, so that the CLI doesn't auto-detect the "current project" and use that to disable various commands, such as xdebug (disabled for non-PHP projects) and drush (for disabled for non-Drupal projects).
  • Locally my machine keeps wanting to remove the latest feature, which is the --quiet (-q) option, from the test help output. I don't understand this.

Follow up steps could be:

  • Build or download PHP and the Legacy CLI on GitHub Actions, so that the integration tests can be run there.
  • Test authenticated commands, complications are:
    • It's hard to test against a known existing project. But it takes a long time (and is expensive) to create a new project.
    • Output becomes less predictable: we could use a template, or just test for specific output with assert.Contains() or assert.Regexp().
    • Performance, and flakiness in general: perhaps these tests should be allow_failure so that they do not block work.
  • Test on multiple operating systems, such as Linux, Windows and MacOS.

pjcdawkins avatar Oct 07 '24 14:10 pjcdawkins