feat: make browsertest failures standout

[dhuang612]

Description of the Change

resolves #792 created a bar that starts out as green when the tests start. Then if one of the tests fails bar will turn red to indicate test failure. From the previous attempt learned that code coverage is decreased. Will work on creating a test for this to get this resolved.

Alternate Designs

Why should this be in core?

Benefits

Allows for less eye movement to notice tests that don't pass when run from google chrome.

Possible Drawbacks

there shouldn't be any side effects because the changes are gui level

changes made are an enhancement.

when the tests first begin:

when a failure is detected:

[coveralls]

Coverage decreased (-0.03%) to 94.378% when pulling 35fd52e4f07327bfa33813535439a078a8cb53cc on dhuang612:make-browsertest-failures-standout into 9e0369b03475d0cc4ffbd32523bdf95b287fe9b7 on mochajs:master.

[dhuang612]

okay, so there aren't tests for the html reporter.

[dhuang612]

reduced the size of the bar and gave them curved edges

success:

failure:

Please let me know if anything else needs to be changed

[pedro-qwkin]

Why isn't this merged?

[dhuang612]

I tried to write some tests under base.spec they complete successfully but don't add to the coverage. Since there are no tests right now for the html browser. Let me know what you want done next.

[juergba]

I recently pinned #3071 in the issue tracker and our exApplitools browser testing crack @giltayar commented: https://github.com/mochajs/mocha/issues/3071#issuecomment-863771039.

I'm aware of that we don't have any html reporter tests, yet. Wouldn't it be much easier to take the path Gil mentioned? A snapshot test with Selenium WebDriver? I'm attracted by Playwright though, I don't know exactly how it works, but it's nice 😎. We could forget about IE11 as we most probably are going to drop its support with the next major release v10. We wouldn't need any mocking, bundling or Saucelab, just a HTML file with a failing test and compare the output against an existing snapshot.

[juergba]

@dhuang612 looks interesting, thank you.

I don't know, why there are no CI tests pending in GH actions to be launched. Can you make a clean rebase, please? You have made some changes to the package.json which are redundant. Can you say anything about the test coverage?

[dhuang612]

@juergba

For test coverage, I got two sites setup one that had failing tests and the other all passing. On the one with the passing, I confirm the bar is green. On the one with failing, I check for the bar to have the .error class added on, and that the color is red.

[juergba]

@dhuang612 I rebased to master and our CI tests have run now.

The coverage has fallen, instead of risen. You have to check in package-scripts.js what's going wrong. When I locally launched npm test a chrome window was opening. There must be a headless option to run these tests.

Edit: there is this message in the CI test log:

something went wrong!
Error: Server terminated early with status 1
    at /home/runner/work/mocha/mocha/node_modules/selenium-webdriver/remote/index.js:248:24
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

[juergba]

I did some testing. The bar is not in the header as the passes, failures, duration fields. If the test run contains many tests and you scroll down the bar will disappear, and those fields remain on top in the header.

[juergba]

First I admit that I don't have any idea of selenium webdriver. Nevertheless I have doubts on these tests here. Requesting https://desolate-sierra-95269.herokuapp.com and the second page is weird. I would have expected a Mocha script with its tests inside, which is run via a server.

The background color of progress-bar.error is always red, so does your test have any meaning at all? Why two progress bars, and not one, where you change the background color accordingly?

[dhuang612]

@juergba "I did some testing. The bar is not in the header as the passes, failures, duration fields. If the test run contains many tests and you scroll down the bar will disappear, and those fields remain on top in the header."

The purpose of the bar is so you know tests fail without having to scroll down, so it doesn't need to always show when scrolling?

[juergba]

No, it don't think so. The color of your bar describes the state of the entire test run, not of a single test. This information (=color) belongs into the header. It can be the color of a bar, or something else eg. the background of the statistic fields.

[dhuang612]

okay, I'll move it into the header, I'll switch the option to get it to run headless, I will change the second test to check for #progress-bar since it starts as green then the class gets added in to make it red. I can check for the existence of the class .error

[dhuang612]

I am having difficulty getting the coverage to run locally from my computer. I researched into how to get this to run according to this SO: https://stackoverflow.com/questions/16633246/code-coverage-with-mocha it should be a test with nyc

But in package-scripts.js

I see a function(test) with the nyc --no-clean

and from my command line when I run npm test(test.node.reporters ) I get a unexpected token '('

The other script I see that mentions nyc is node

but running npm test.node doesn't give me any coverage information.

What am I missing?

Thank you for your help!

[juergba]

I have asked someone for help, to setup our first HTML reporter browser tests. I don't have time to get into this. So please focus on the core issue of this PR, maybe we will shortly have some HTML reporter test which then can easily be extended.

[dhuang612]

how it looks with passing tests tests with failures

[juergba]

@dhuang612 I got a negative reply by the person I contacted, so the browser tests remains unsolved. He recommended to use playwright. I don't know yet how to proceed.

Your screenshots look nice, the colors look evtl. a bit strong. I haven't tested, yet. #mocha-stats.fail: is there no other way like extend, more elegant than repeating the entire #mocha-stats? Is CSS so ugly?

[dhuang612]

I'll take a look at using extend.

[dhuang612]

@juergba I added in Sass as a dev package. I then update the css file name to sass. Finally, I switched the class I added to extend the original, and only updated the status bar colors themselves.

I tested the feature out using the example files to make sure they worked. I also squashed the extra commit.

The tests were failing because the files were expecting .css I updated them to sass. I can update the documents after confirming this is the directions we want to take