bug: browserHeadless: "new" resume in "App did not load in allowed time."

[danyball]

Prerequisites

• [X] I have read the Contributing Guidelines.
• [X] I agree to follow the Code of Conduct.
• [X] I have searched for existing issues that already report this problem, without success.

Stencil Version

4.0.4

Current Behavior

When running many tests/test suits in parallel the error occurs. Without browserHeadless: "new" the error do not occur.

Error output:

  ● my-component › renders

    App did not load in allowed time. Please ensure the content loads a stencil application.

       5 |     const page = await newE2EPage();
       6 |
    >  7 |     await page.setContent('<my-component></my-component>');
         |     ^
       8 |     const element = await page.find('my-component');
       9 |     expect(element).toHaveClass('hydrated');
      10 |   });

      at waitForStencil (node_modules/@stencil/core/testing/index.js:834:9)
      at e2eSetContent (node_modules/@stencil/core/testing/index.js:824:9)
      at Object.<anonymous> (src/components/my-component/3my-component.e2e.ts:7:5)

Expected Behavior

No error.

System Info

MacOS, log: `jest args: --e2e --max-workers=8`,

      System: node 20.5.1
    Platform: darwin (22.6.0)
   CPU Model: Apple M1 Max (10 cpus)
    Compiler: /Users/myownfolder/stencil-component-starter/node_modules/@stencil/core/compiler/stencil.js
       Build: 1691405536
     Stencil: 4.0.4 🍧
  TypeScript: 5.0.4
      Rollup: 2.42.3
      Parse5: 7.1.2
      Sizzle: 2.42.3
      Terser: 5.19.2

Steps to Reproduce

This is what I have changed in your example repo: https://github.com/danyball/stencil-app-timeout/commit/668870a037a3420e90b77c9cd0a83db1e9a9710c

1. clone that repo
2. run "stencil test --e2e"

Code Reproduction URL

https://github.com/danyball/stencil-app-timeout.git

Additional Information

There's a thread in Discord: https://discord.com/channels/520266681499779082/1113011702259269672

[tanner-reits]

Hey @danyball 👋

Thanks for reporting this issue and for the reproduction! I have confirmed this is an issue and done some initial investigation. We haven't yet pinpointed if this is an issue within Stencil, or one of our testing dependencies (Jest or Puppeteer), but I have identified that the behavior can be changed by modifying the --max-workers argument that gets passed through to the Jest CLI.

So, if you are looking for a temporary workaround without needing to revert to the old headless behavior, you can run one of the following commands:

stencil test --e2e --runInBand

// or

stencil test --e2e --max-workers=1

There will obviously be performance drawbacks as tests won't run in parallel, but I haven't encountered the timeout issue at all with either of those commands. I've also had limited success if I set the number of max-workers to a value less than the number of test suites to be ran, but it is inconsistent. This aligns, however, with the finding that the tests may pass running in watch mode since Jest defaults the number of workers to half the number of cores available on your machine (associated Jest docs).

We'll get this issue labeled for us to continue investigating and hopefully have a better solution soon! Thank you for your patience!

[danyball]

Is nobody using browserHeadless: "new" or parallel test runs? Hoped that this issue gets some priority :)

[narndt-inform]

Hi @danyball , we are using the option and are experiencing the same issues as you described. We made them a bit more bearable in our CI by only using 1 worker and retrying the job twice when it fails. This reduced our false negative runs to ~2%. Without the retries (which of course take up CI time), we are at a false negative failure rate of >60% (especially with 2 or more workers). So yeah, we've found a more or less reliable workaround but would of course also love to see the issue gone :)

We've also noticed that allocating more resources (CPU) to the test runner works to reduce the failure rate. Maybe this can help when trying to reproduce the issue more reliably, or for working around the issue for now!

[danyball]

Thanks. Just using 1 worker Is not an option for us because we have too many tests. But we are also using retry :).

We need the fix especially for "screenshot testing". Because taking screenshots on a local mac are bit different to those on CI server. But activating "headless new" will fix this.

[SvenKauer]

We have the same Issue, and it´s also not working with maxWorkers=1. The tests fail constantly in every test run. Setting a higher timeout has no effect. Is there another workaround we can try?

[christian-bromann]

@SvenKauer can you produce a minimal reproducible example for this behavior?

[SvenKauer]

Unfortunately, I cannot say what´s the problem. The bug came with the upgrate to Version 22 of puppeteer, even in e2e tests that are just test if they have where hydrated like the first example from danyball. Furthermore it´s maby helpful to know, that we don´t have browser tests, we use puppeteer for clicking-tests.

[christian-bromann]

that we don´t have browser tests, we use puppeteer for clicking-tests.

How do you define both types of tests?

Unfortunately, I cannot say what´s the problem.

Without a reproducible example this would be difficult for us to look into and we would have to close the issue.

[SvenKauer]

A test looks like this: import { newE2EPage } from '@stencil/core/testing';

describe('checkbox', () => { it('renders', async () => { const page = await newE2EPage({ timeout: 10000 }); await page.setContent('');

const element = await page.find('checkbox');
expect(element).toHaveClass('hydrated');

}); )};

Simple and no specifics in it.

[christian-bromann]

await page.setContent('');

What is the reason for setting an empty content?

[SvenKauer]

No, I don´t know why github shows this like this, if I wanted to edit it and I see this:

So, the content is not empty, the content is < checkbox ></ checkbox >- (bitbucket seems to remove tags in the view...)

[marksy]

@SvenKauer

So, the content is not empty, the content is < checkbox ></ checkbox >- (bitbucket seems to remove tags in the view...)

you need to use a code block or surround the code with ``` at the beginning and the end

[christian-bromann]

Hey you all 👋 at this point I recommend to use alternative testing frameworks for testing Stencil components (either e2e or as component test). Please have a look:

• https://stenciljs.com/docs/testing/webdriverio/overview
• https://stenciljs.com/docs/testing/playwright/overview

Our team has limited resources and probably won't be able to look into this soon. We however would review and merge any PRs from the community.