CT Webpack 5 projects don't surface compilation errors/warnings

[ZachJW34]

Current behavior

After an initial clean build (no compilation errors), compilation errors are no longer surfaced in the runner. Editing a spec file to throw a compilation error causes the app to reload but with no visual indicator that the compilation failed (and the tests will even rerun). You can see the errors in the console and the terminal, but users aren't always paying attention to these.

https://user-images.githubusercontent.com/25158820/183713613-860e9594-eff7-4544-a70a-d1e29a47cd94.mov

Desired behavior

The user should be made aware of the compilation error in the runner, otherwise they may be led to believe that their test is passing or their changes are being reflected.

Test code to reproduce

I was only able to reproduce this with certain Webpack 5 projects, such as a Next.js application and our in progress Angular work. CreateReactApp v5 doesn't exhibit this behavior

1. npx create-next-app my-next-app && cd my-next-app
2. npm i -D cypress
3. yarn cypress open
4. Walk through CT setup and launch a browser
5. Create pages/home.cy.js with contents:

import Home from "./index";

it("should render", () => {
  cy.mount(<Home />);
});

6. Run test in Cypress
7. Modify pages/home.cy.js so that it will throw a compilation error, notice the app refreshes and test runs even though spec is invalid.

Cypress Version

10.4.0

Other

The issue seems to be related to Webpack's caching feature, and more complicated frameworks (such as Next.js) have more sophisticated caching mechanisms. This behavior is inline with normal dev experience, the only difference is that most frameworks utilize an overlay to display the error. We axed the overlay in #21599, but it would prove useful in this scenario. There are some downsides of enabling the overlay, since it will display over the AUT for any compilation errors/warnings, even if the error isn't related to the current test.

A more robust fix would be to disable the dev-server from serving files that have compilation errors. My thoughts are that since there is a cached version of the module, it is getting served in the broken modules place.

[baus]

Hey team! Please add your planning poker estimate with ZenHub @astone123 @lmiller1990 @marktnoonan @mike-plummer @rockindahizzy @warrensplayer @ZachJW34

[lmiller1990]

A more robust fix would be to disable the dev-server from serving files that have compilation errors. My thoughts are that since there is a cached version of the module, it is getting served in the broken modules place.

What about the error - we should still surface this, right? Eg - it would not matter if we serve a stale/cached version, as long as the error is served in the overlay on top of the AUT.

I think we've got some tests for this, evidently not for projects with Next.js.

[lmiller1990]

We've got a pattern for this in webpack preprocessor - try making app using webpack pre-processor with errors, it should manifest in the iframe. We could do the same thing here.

[ZachJW34]

@lmiller1990 There is some information in the ct-architectural-improvements, from what I understand we stopped showing the overlay (which would clearly point out what's going wrong) due to errors occurring in unrelated specs still causing the overlay to pop up (it's still a single compilation so an error compiling faling.cy.js will affect passing.cy.js).

Could we compromise and show the overlay in open mode and disable it for run mode?

[lmiller1990]

I see, we probably need to figure out what "classes" of errors exist and how each one should manifest. I think open mode and run mode should be as similar as possible - if there's a compilation error on CI, I think you'd want to see that in the video, right (since you can't see the console there).

due to errors occurring in unrelated specs still causing the overlay to pop up

Any concrete example for this?

[rockindahizzy]

Investigate script error (e2e flow for webpack preprocessor)

[cypress-bot[bot]]

The code for this is done in cypress-io/cypress#23844, but has yet to be released. We'll update this issue and reference the changelog when it's released.

[cypress-bot[bot]]

Released in 10.9.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to Cypress v10.9.0, please open a new issue.