Multi PHP + Multi Environment + Check all commits

[javiercasares]

This is the Cloudfest Hackathon 2024 PR.

This PR solves:

• Multi PHP. There is a new env option where you can set all the different PHP versions you want to test.
• Multi Environment. You can set a label to differentiate, for example, your “shared”, “vps”, or “cloud” setups.
• Check all commits. Until now, only the latest commit can be checked. With this new option, we can check the latest 10 commits so none of the commits are missing if they are approved without a lot of time between them.

The test-reporter will be upgraded to comply with these changes.

[javiercasares]

FYI: https://github.com/WordPress/phpunit-test-reporter/pull/109

[javiercasares]

The code is not very beauty, but it should work...

[javiercasares]

I applied the suggestion @pfefferle did on WordCamp Europe, and reduce a lot of duplicated code that we couldn't do at the Cloudfest Hackathon, mostly on the parts when testing multiple PHP versions.

I did some improvements on the documentation, with some scripts so the software (from the main branch) is updated and maintained automatically.

I've been doing some tests in my local machine, and on my production server (where I was doing the former tests) and looks like everything is working.

On my side, I think everything is in order, and we can approve this and put it available for everyone.

[benniledl]

Running tests in parallel?

Running the PHPUnit test suite across 7 different PHP versions is taking a long time in my environment. Is there a way to run the tests in parallel to speed things up?

Preparing a new environment for every version necessary?

Also, is it necessary to create a completely new environment for each PHP version? So far, I've been running the prepare.php file for the first PHP version, running the tests then report.php then using a script to search and replace the PHP executable in the configuration files before rerunning the tests and reporting again. This approach seemed to work.

Copying the environment + Running in parallel

I could see that using the same environment for multiple php versions in parallel could cause problems, is it maybe possible to just copy one environment to a different directory and then replace the php version in the config? Since preparing an environment also takes a long time for me.