Consider using GitHub actions to run Drush tests

[greg-1-anderson]

Is your feature request related to a problem? Please describe. Windows testing on Appveyor has always been a little unreliable; Appveyor only supports one version of PHP, and there's always a chance that tests will break when they upgrade to a newer release. I was recently spurred to look into GitHub actions more closely because Travis queue times have been getting longer and longer -- but that was for another project, Drush has been Travis-free for some time.

Describe the solution you'd like Drush is already using GitHub actions for building documentation pages; expanding our use of it to the tests would give more consistency.

Advantages of GitHub actions:

• Speedy
• Support for Windows in the same config file as Linux
• Easy matrix builds, like Travis
• If someone forks your repo, they get a working copy of your active CI configuration as well (IDK if that matters for us, but it sounds pretty cool)
• Cool backward-compatability test (OK this one is not limited to GitHub actions, it's roave/backward-compatibility-check, and we could do it in Circle too -- I just discovered it while looking at GitHub action config files)

Disadvantages:

• Can't ssh into the test container
• Limited ability to syntax-check config file; there is a VSCode plugin, and GitHub has published the js-schema for it, but that hasn't been made available in a CLI tool, at least not that I have found. If there's an error in your config, your tests do not run and there is no feedback.

Describe alternatives you've considered We can stay as we are.

Additional context I have a working GitHub actions config file in https://github.com/consolidation/output-formatters/pull/88/files. Also shows testing on phpunit 7 thru 9 with yoast/phpunit-polyfills (supports testing on PHP 7.1 - 8.0).

[weitzman]

I'd say I'm neutral on the proposal. The 'SSH into container' feature is very helpful on occasion. Also, how many concurrent builds do we get? Circle gives us 4.

Thanks for pointing out the roave and yoast packages. Interesting.

I agree that moving the Consolidation libraries to Github Actions is a good first step.

[greg-1-anderson]

It looks like Travis is no longer turned on for drush/drush, so maybe I will switch to GitHub actions for Drush 8 after I do the Consolidation projects. Doing that will require me to figure out how to mix phpunit <= 6 with phpunit >= 7 (or maybe just stick with <= 6 and say goodbye to php8).

I poked around a little bit and couldn't find any documentation on how many concurrent builds are allowed for public / unpaid projects.

[greg-1-anderson]

Looks like there's a solution to the setup() problem in the README:

https://github.com/Yoast/PHPUnit-Polyfills/blob/701f40c45a22f36906016a0aa019ebd72b966edb/README.md#testcases

[greg-1-anderson]

Apparently it is possible to ssh into a GitHub actions container using https://github.com/mxschmitt/action-tmate. Have not tried it yet.

[greg-1-anderson]

Close this if you wish; see #4598

[weitzman]

I'll leave it open as a feature request in case Windows DBs become easier on Github Actions. Thanks for your efforts, @greg-1-anderson

[greg-1-anderson]

The other thing I wanted to record here is that I did try roave/backward-compatibility-check in #4598, but it was way too picky for Drush and flagged things we wouldn't really consider to be a b/c break (e.g. adding another commandline option to an $options array default value). We might be able to configure it to ignore such things, but it is questionable whether the tool would still have enough value if we did that.

[weitzman]

Looks like Github offers an impressive 20 concurrent jobs. I'm once again contemplating moving. CircleCI just started limiting us to 1 concurrent. I emailed their Support because I think its a mistake but even if thats reverted, GH is looking attractive.