coderoad-vscode icon indicating copy to clipboard operation
coderoad-vscode copied to clipboard

Fix test runner failure on PHP tutorials

Open jcandan opened this issue 10 months ago • 3 comments

Describe the bug

Getting a Test Runner Failed message on a PHP project.

To Reproduce

Requires PHP 8.2+ and Composer.

Steps to reproduce the behavior:

  1. Open an empty folder in VS Code
    mkdir empty-project1
    code empty-project1
    
  2. Run CodeRoad: Start from the Command Pallete.
  3. Hit Start New Tutorial
  4. Choose URL, and supply https://raw.githubusercontent.com/jcandan/learn-php-basics/refs/heads/main/tutorial.json
  5. Open terminal, and in this empty project folder, run git init.
  6. Hit Start on the tutorial
  7. Supply an empty index.html file, and hit Run

See the Test Runner Failed message.

Expected behavior

To see that the solution should have passed:

  1. Clone the https://github.com/jcandan/learn-php-basics project locally
  2. Checkout the v0.1.0 branch, you'll be on the 1.1S solution commit
  3. Run cd .stack-elevate/ && composer install && ./vendor/bin/phpunit to prove the test runner should have worked

The TAP output shows that the test does pass.

Desktop (please complete the following information):

  • Platfrom: MacBook Pro
  • OS: macOS Sequoia
  • Version 15.3

Node:

  • Version v20.16.0

Git:

  • Version 2.40.0

PHP:

  • Version 8.4.3

Composer:

  • Version 2.8.5

PHPUnit:

  • Version 11.5.6
  • nikeee/phpunit-tap TAP Package required by INIT commit
  • The phpunit.xml configuration ensures the the TAP output
  • The config.testRunner.args.tap required configuration is blank since it is handled by the XML configuration.

jcandan avatar Feb 09 '25 23:02 jcandan

The nikeee/phpunit-tap outputs TAP compliant reporting. However, it currently does include PHPUnit output as well. I suspect this may be causing the test runner to fail.

According to nikeee/phpunit-tap/issues/31,

having non-TAP-lines between the TAP output is not a problem according to the TAP spec

In that case, the consumer in CodeRoad VS Code extension should be able to parse the combined PHPUnit and TAP compliant output.

However, I have responded to that claim to say that the TAP output stream should be entirely valid TAP. Awaiting his response.

But, in the interrim, we might want to at least confirm via debugging that this is the reason for the test runner failure.

jcandan avatar Feb 09 '25 23:02 jcandan

Okay, https://github.com/nikeee/phpunit-tap/issues/31#issuecomment-2646650518 makes a fairly good case that TAP should be able to deal with other output. So, if we do find this to be the cause of the failure, perhaps the TAP handling should be updated to deal with that.

jcandan avatar Feb 10 '25 00:02 jcandan

I've updated the example https://github.com/jcandan/learn-php-basics project with a patch from https://github.com/nikeee/phpunit-tap/pull/32 to ensure the first line of the TAP output is on a new line. So, we definitely should be able to run the test without failure.

Also, I noticed that the Run click returns the Test Runner Failed message fairly quickly. So, I don't think it is even getting to the point of reading the TAP output anyway.

Would really appreciate eyes on this one. Thanks!

jcandan avatar Feb 10 '25 00:02 jcandan