Get functional test suite working on GitHub Actions

[colinmollenhour]

Goal: get the Magento Test Framework tests working via GitHub Actions so we actually have a robust functional test suite.

[colinmollenhour]

Made a lot of progress but hit a wall:

https://travis-ci.org/github/OpenMage/magento-lts/jobs/683167264

PHP Fatal error: Declaration of Magento\Mtf\TestSuite\TestCase::run(?PHPUnit_Framework_TestResult $result = NULL) must be compatible with Magento\Mtf\TestSuite\TestSuite::run(?PHPUnit_Framework_TestResult $result = NULL, $filter = false, array $groups = Array, array $excludeGroups = Array, $processIsolation = false) in /home/travis/build/OpenMage/magento-lts/dev/tests/functional/vendor/magento/mtf/Magento/Mtf/TestSuite/TestCase.php on line 126 Fatal error: Declaration of Magento\Mtf\TestSuite\TestCase::run(?PHPUnit_Framework_TestResult $result = NULL) must be compatible with Magento\Mtf\TestSuite\TestSuite::run(?PHPUnit_Framework_TestResult $result = NULL, $filter = false, array $groups = Array, array $excludeGroups = Array, $processIsolation = false) in /home/travis/build/OpenMage/magento-lts/dev/tests/functional/vendor/magento/mtf/Magento/Mtf/TestSuite/TestCase.php on line 126

This code is not in our repo, though.. It is in magento/mtf.

[colinmollenhour]

Latest progress: https://travis-ci.org/github/OpenMage/magento-lts/builds/683476886

[colinmollenhour]

I got it to execute PHPUnit on PHP 7.2 with no errors but for some reason no tests were actually run.. https://travis-ci.org/github/OpenMage/magento-lts/jobs/683514414

I don't know anything about the MTF framework. Does anyone else have any clues as to how to get this to run?

[colinmollenhour]

Runs on PHP 7.2, 7.3 and 7.4, but times out with no details about what errors occurred... https://travis-ci.org/github/OpenMage/magento-lts/jobs/683573694

I suppose this would need to be run locally to debug it further.. :(

[tmotyl]

for the record, here is the instruction for the old mtf framework for m1. https://web.archive.org/web/20180303132243/http://devdocs.magento.com/guides/v2.0/mtf/mtf_quickstart/mtf_quickstart_config.html

If we find it useful we should fork it in openmage and adapt to suite our needs.

[tmotyl]

when I try to run

sh docker/run.sh

I'm getting

+ docker-compose -p mtf up -d
ERROR: 
        Can't find a suitable configuration file in this directory or any
        parent. Are you in the right directory?

        Supported filenames: docker-compose.yml, docker-compose.yaml

[tmotyl]

I've took the liberty to rebase the branch on top of current 1.9.4.x and add a commit with executable permissions for sh scripts

[colinmollenhour]

Tests are still failing, I think it is unable to login for some reason. I changed the password to match but may have missed something...

https://travis-ci.org/github/OpenMage/magento-lts/jobs/700745572

[fballiano]

it's been more than 2 years without movement on this PR, I'll close it but can always be reopened in case somebody want to continue it.

[github-actions[bot]]

Unit Test Results

7 tests   5 :heavy_check_mark:  0s :stopwatch: 1 suites  2 :zzz: 1 files    0 :x:

Results for commit 46e5c36c.

[github-actions[bot]]

Unit Test Results

7 tests   7 :heavy_check_mark:  0s :stopwatch: 1 suites  0 :zzz: 1 files    0 :x:

Results for commit b5df99e5.

:recycle: This comment has been updated with latest results.

[sreichel]

@colinmollenhour follow your commits ... absolutly awesome :heart:

[colinmollenhour]

Sooo close... https://github.com/OpenMage/magento-lts/actions/runs/3890167666/jobs/6639043916#step:16:18

EDIT: The Apache test requests are failing but do not trigger an error so I think that is actually the issue.. I'm not familiar with mod_proxy_fcgi so probably did something wrong there.

[colinmollenhour]

Sadly I must give up on this for a while.. If anyone wants to help troubleshoot, please be my guest!

• Apache is responding with 403/503 so there is some configuration issue. This probably fully explains the "Admin user cannot be logged in by curl handler!" but that error should be replaced with something more useful.
• There are some deleted files from merging latest 1.9.4.x - I don't know why they were deleted or if they should have been?
• It is running on PHP 7.3 currently - 8.1.14 is broken woth timezone issue - also probably upgrading PHPUnit will break some stuff and the MTF code may need to be updated to fix code broken by 7.4, 8.0, 8.1
• No idea if the functional tests will still pass (but they should?), just trying to get them "running" again.. the code appears to have reached the point where it logs into the admin panel so it would seem that it is working to some degree although I don' tknow if this is actually talking to Selenium yet.

[elidrissidev]

Invading to try something 🕵️

[elidrissidev]

I think we have some progress. 503 error is substituted by 403 when requesting http://localhost/index.php

[sreichel]

I've no time at the moment ... can you try http://127.0.0.1/ instead?

[colinmollenhour]

Rebased onto main. These files were deleted in upstream, I am not sure why or if it is ok?

Unmerged paths:
  (use "git restore --staged <file>..." to unstage)
  (use "git add/rm <file>..." as appropriate to mark resolution)
        deleted by them: dev/tests/functional/lib/Magento/Mtf/App/State/AbstractState.php
        deleted by them: dev/tests/functional/lib/Magento/Mtf/App/State/State1.php
        deleted by them: dev/tests/functional/lib/Magento/Mtf/Client/Driver/Selenium/Driver.php
        deleted by them: dev/tests/functional/lib/Magento/Mtf/Client/Element/SimpleElement.php
        deleted by them: dev/tests/functional/lib/Magento/Mtf/Util/Generate/Page.php

[fballiano]

question: does it still make sense in 2024 to keep this open?

[Flyingmana]

getting the functional testsuite to work again would at least increase our coverage a lot lot. The alternative would be to build up or own for all these cases

[colinmollenhour]

I'm on the fence.. It would be great to leverage the existing work but at the same time it seems kinda ancient and klunky.. I'd much rather use something like Cypress (I'm sure there are many other good choices) but we'd be starting from scratch. As far as my existing work on this PR, if it was for nothing that is ok - no sunk cost bias here.

I may know a company that would be willing to sponsor the development of a modern test suite if it could be done fairly fast (weeks or months, not years). Is anyone interested in that opportunity?

[mattdavenport]

Any objections to closing this PR?

[sreichel]

Cypress lgtm but its not free. What alternatives we have? Selenium? ...?

[mattdavenport]

Cypress has a free MIT-licensed version. I think it's when you get into parallel pipeline running and UI features that it's paid. There are also some ways around this as well, but the open source version has been sufficient for us.

[colinmollenhour]

There is also Playwright. From what I've seen, Cypress and Playwright are the best choices in my opinion. The free version of Cypress is definitely sufficient for our purposes.