test(html): add playwright html test

[BreadGenie]

fix #1911

[BreadGenie]

Almost all the tests failing are runtime errors. I think maybe HTML tests are screwing something up in the CI (It worked on my machine :P ). Should I run HTML tests separate to sync and async tests?

[BreadGenie]

@terriko I don't think it's the same errors. I ran tests on my fork for both main and this branch and the tests ran on main seems to have passed without any issues. But once there are playwright tests present the runtime issues started occurring.

The html test failure from last commit (cccd678ec9c138ee5b9fbfe39c32fa61de65a468) was a fault from my side since I installed firefox instead of chromium for the tests

[BreadGenie]

The test failures are all runtime errors

=========================== short test summary info ============================
FAILED test/test_strings.py::TestStrings::test_curl_7_34_0 - RuntimeError: Ca...
FAILED test/test_strings.py::TestStrings::test_kerberos_1_15_1 - RuntimeError...
FAILED test/test_version.py::TestVersion::test_different_version[-1] - Runtim...
FAILED test/test_version.py::TestVersion::test_different_version[1] - Runtime...
FAILED test/test_version.py::TestVersion::test_exception - RuntimeError: Cann...
===== 5 failed, 384 passed, 322 skipped, 120 warnings in 84.96s (0:01:24) ======

interestingly the error says

f"The test {pyfuncitem} is marked with '@pytest.mark.asyncio' "
"but it is not an async function. "
"Please remove asyncio marker. "
"If the test is not marked explicitly, "
"check for global markers applied via 'pytestmark'."

which is true for all tests in test_version.py but not for test_version.py

[BreadGenie]

When I separated the html tests all of them passes fine (gh action run). I'll look more into test_version.py and test_version.py see if I can do something about it.

Edit 1: I was able to fix test_version.py test errors by simply removing the async keyword and asyncio markers. Edit 2: Refactored test_strings.py without async code (https://github.com/BreadGenie/cve-bin-tool/commit/4fe7e5d060a1425c6e1f7748ed7b633111aff8d3) but still fails in CI, but passes on local machine (gh action run).

[codecov-commenter]

Codecov Report

Merging #1925 (9f99e22) into main (b4d2f26) will decrease coverage by 1.60%. The diff coverage is 15.78%.

@@            Coverage Diff             @@
##             main    #1925      +/-   ##
==========================================
- Coverage   79.91%   78.31%   -1.61%     
==========================================
  Files         547      549       +2     
  Lines        8685     8766      +81     
  Branches     1013     1017       +4     
==========================================
- Hits         6941     6865      -76     
- Misses       1473     1629     +156     
- Partials      271      272       +1     

Flag	Coverage Δ
longtests	78.31% <15.78%> (-1.61%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
test/pages/html_report.py	0.00% <0.00%> (ø)
test/test_html.py	0.00% <0.00%> (ø)
test/test_output_engine.py	96.47% <ø> (-0.82%)	:arrow_down:
cve_bin_tool/version_scanner.py	88.19% <90.47%> (+0.25%)	:arrow_up:
cve_bin_tool/cli.py	66.89% <100.00%> (+1.25%)	:arrow_up:
cve_bin_tool/output_engine/html.py	59.61% <0.00%> (-28.85%)	:arrow_down:
cve_bin_tool/output_engine/print_mode.py	79.31% <0.00%> (-10.35%)	:arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[terriko]

This is looking good, would you say it's ready to merge?

For me: playright license is Apache, pytest-playwright license is also Apache. Unlikely to be any license compatibility issues but I'll go file the required licensing paperwork before I merge.

[BreadGenie]

Yep, it's ready to be merged. I've included test cases that I've often broke while making changes, but there's a lot of room for improvement.

[terriko]

(going to let the tests run again before merge since it's been a while and I had to resolve a dependency conflict)

[BreadGenie]

Not sure why one test failed. Tests on fork seems to have passed.

[terriko]

I'm going to re-run just that one. The fact that it's only happening on 3.7 makes me wonder if it's related to the same problems we've been having with other mocked file tests.

[terriko]

Oh and I'll add that the license compatibility paperwork is approved, so CI is the only thing I'm waiting on now.