[Bug Found + Fix] Full Stack Code Coverage

[flogh]

I'm submitting a ...

Bug report plus solution for improvement.

Context

I've decided to implement the fullstack code coverage on my project which is a front Angular with back Nestjs. It's quite a big project, both in the front and the back side.
Implementation with Angular was really straightforward thank's to https://github.com/skylock/cypress-angular-coverage-example

Fullstack Implementation was simple too, but led me to some bugs. The cy.request() made during the afterAll hook which is supposed to get the coverage json was failing like 9 out of 10 times (timeout error). I quickly deduced that the size of the json sent was the problem, it's about 5mb.

I tried multiple things like, changing the response timeout to 5 minutes, sending the result compressed as gzip, or sending the json as a string, then parsing it in the front, but none of these solutions worked.

And then I saw that under the hood cy.request which is used to communicate with the backend to get the coverage json, was used with HTTP1 which is super slow.
So I found this workaround which now gets the json, in like 1 sec instead of 20sec (when it worked).

Current implementation

// node_modules/@cypress/code-coverage/support.js

after(() => {
    // there might be server-side code coverage information
    // we should grab it once after all tests finish
    const baseUrl = Cypress.config('baseUrl') || cy.state('window').origin
    const runningEndToEndTests = baseUrl !== Cypress.config('proxyUrl')
    if (runningEndToEndTests) {
        // we can only request server-side code coverage
        // if we are running end-to-end tests,
        // otherwise where do we send the request?
        const url = Cypress._.get(
            Cypress.env('codeCoverage'),
            'url',
            '/__coverage__'
        )
        cy.request({
            url,
            log: false,
            failOnStatusCode: false
        })
        .then(r => Cypress._.get(r, 'body.coverage', null), { log: false })
        .then(coverage => {
            if (!coverage) {
                // we did not get code coverage - this is the
                // original failed request
                return
            }
            sendCoverage(coverage, 'backend')
        })
    }
    ...

Workaround

// node_modules/@cypress/code-coverage/support.js

after(() => {
    // there might be server-side code coverage information
    // we should grab it once after all tests finish
    const baseUrl = Cypress.config("baseUrl") || cy.state("window").origin;
    const runningEndToEndTests = baseUrl !== Cypress.config("proxyUrl");
    if (runningEndToEndTests) {
        // we can only request server-side code coverage
        // if we are running end-to-end tests,
        // otherwise where do we send the request?
        const url = Cypress._.get(Cypress.env("codeCoverage"), "url", "/__coverage__");
        /* HERE IS THE FIX */
        cy.exec(`curl ${url}`).then(r => {
            if (!r || !r.stdout) {
                // we did not get code coverage - this is the
                // original failed request
                return;
            }
            const coverage = JSON.parse(r.stdout).coverage;
            sendCoverage(coverage, "backend");
        });
        /* END OF FIX */
    }
    ....

I wasn't sure about the coding style that you have, that's why I'm opening an issue first and not a PR. Let me know if you want me to try something :)

By the way you are doing an amazing job at cypress ! 👏

PS

Actually, before implementing the full stack code coverage on my big project, I've first made a minimal working repository with angular nestjs and cypress, to figure out how the implementation could be done.

Here is the link : https://github.com/flogh/cypress-angular-nestjs-code-coverage-fullstack

Maybe you'll want to add it to with the other implementation examples.

[bahmutov]

Hmm, I see, the backend code coverage object is giant and cy.request fails to fetch it quickly or at all.

I don't think we can just exec curl in this case, but you did give me an idea. Why do we need to fetch the backend code coverage? We fetch it from the test (in the browser) and then immediately send it back to the plugin using cy.task to be merged with the rest of the code coverage information.

The coverage goes from the backend to the browser to be immediately sent back to the backend! I think a better solution here would be to fetch coverage in the plugins file using Node's request, which should be fast and cross-platform. Let me implement this as a solution. Also, thanks a lot for the project link

[bahmutov]

I cloned your repo, but when trying to start it I am getting

[start:nestjs] 
[start:nestjs] > [email protected] start:coverage /Users/gleb/git/cypress-angular-nestjs-code-coverage-fullstack/nestjs
[start:nestjs] > nyc --silent nest start
[start:nestjs] 
[start:nestjs] sh: nyc: command not found
[start:nestjs] npm ERR! code ELIFECYCLE
[start:nestjs] npm ERR! syscall spawn
[start:nestjs] npm ERR! file sh
[start:nestjs] npm ERR! errno ENOENT
[start:nestjs] npm ERR! [email protected] start:coverage: `nyc --silent nest start`
[start:nestjs] npm ERR! spawn ENOENT
[start:nestjs] npm ERR! 
[start:nestjs] npm ERR! Failed at the [email protected] start:coverage script.
[start:nestjs] npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

You should probably add nyc as a dependency

[flogh]

The thing is that even after installing nyc as a dependency, it's still unavailable through command line. That's why I suggested at the beginning of the readme to install it globally :

$ npm i -g  @angular/cli @nestjs/cli cypress nyc

But you're right, maybe I should put it in the dependencies and change the nestjs start command from

nyc --silent nest start

to

./node_modules/.bin/nyc --silent nest start

What do you think ?

[bahmutov]

Should be fine in the root dev dependencies of the project

Sent from my iPhone

On Feb 11, 2020, at 10:21, Flow [email protected] wrote:

 The thing is that even after installing nyc as a dependency, it's still unavailable through command line. That's why I suggested at the beginning of the readme to install it globally :

$ npm i -g @angular/cli @nestjs/cli cypress nyc But you're right, maybe I should put it in the dependencies and change the nestjs start command from

nyc --silent nest start to

./node_modules/.bin/nyc --silent nest start What do you think ?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe.

[flogh]

Ok, done :) Nyc has to be installed globally anyway to generate the /coverage folder from the .nyc_output/

[bahmutov]

Well you could install it locally and just do “npx nyc ...” to generate the reports

Sent from my iPhone

On Feb 11, 2020, at 12:42, Flow [email protected] wrote:

 Ok, done :) Nyc has to be installed globally anyway to generate the /coverage folder from the .nyc_output/

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe.

[accordionpeas]

Hi,

Is there any update on this? I'm seeing the same issue, the collectBackendCoverage hook times out when the JSON response is too large.

Thanks,

[nickpalmer]

We are also experiencing this, with failure on maybe 1 out of 10 pulls. @bahmutov any progress on an official solution to this issue?

[dineshwarandh]

@bahmutov Do we have any update on this issue? We are also facing the same issue, the backend coverage object is ~20MB in our case. Able to get the response when the endpoint is accessed via Postman/ cURL command, but it is timed out in cy.request().

[bahmutov]

I think cypress team would welcome a pull request

[idrisjaffer89]

Any update/workaround on this issue? In my case, the coverage payload is 7 MB, and Cypress always times out.

[IdrissMahjoubi]

@idrisjaffer89 try sendCoverage(JSON.stringify(coverage), 'backend')