Cypress reporter error in 10.7.0+ in Azure Pipileines with CRA

[paulgrant]

Current behavior

After ugrading cypress from 8 to 10.8.0 we are experience flaky component tests with the reporter reporting null issues. We have tried 10.9.0 (hung completly) and 10.7.0 experiencing the same.

We are using CRA with cypress, babel, ny, istanbul, cypress/code-coverage

Invalid array length
RangeError: Invalid array length
    at Runner.runner._events.pass.<computed> (/home/vsts/.cache/Cypress/10.8.0/Cypress/resources/app/packages/server/lib/reporter.js:314:11)
    at Runner.emit (node:events:538:35)
    at Runner.emit (node:domain:475:12)
    at Reporter.emit (/home/vsts/.cache/Cypress/10.8.0/Cypress/resources/app/packages/server/lib/reporter.js:362:46)
    at Object.onMocha (/home/vsts/.cache/Cypress/10.8.0/Cypress/resources/app/packages/server/lib/project-base.js:272:34)
    at Socket.<anonymous> (/home/vsts/.cache/Cypress/10.8.0/Cypress/resources/app/packages/server/lib/socket-base.js:260:40)
    at Socket.emit (node:events:526:28)
    at Socket.emit (node:domain:475:12)
    at Socket.emitUntyped (/home/vsts/.cache/Cypress/10.8.0/Cypress/resources/app/packages/socket/node_modules/socket.io/dist/typed-events.js:69:22)
    at /home/vsts/.cache/Cypress/10.8.0/Cypress/resources/app/packages/socket/node_modules/socket.io/dist/socket.js:428:39
    at processTicksAndRejections (node:internal/process/task_queues:78:11)

Desired behavior

The tests should run and report on code coverage as previously.

Test code to reproduce

The config is here:

import { defineConfig } from 'cypress';

export default defineConfig({
  viewportHeight: 768,
  viewportWidth: 1024,
  video: false,
  defaultCommandTimeout: 12000,
  requestTimeout: 12000,
  retries: {
    // Configure retry attempts for `cypress run`
    runMode: 2,
    // Configure retry attempts for `cypress open`
    openMode: 0,
  },
  component: {
    specPattern: 'src/**/*.spec.{js,jsx,ts,tsx}',
    setupNodeEvents(on, config) {
      // eslint-disable-next-line @typescript-eslint/no-var-requires
      require('@cypress/code-coverage/task')(on, config);
      return config;
    },
    devServer: {
      framework: 'create-react-app',
      bundler: 'webpack',
      webpackConfig: {
        mode: 'development',
        devtool: false,
        module: {
          rules: [
            // application and Cypress files are bundled like React components
            // and instrumented using the babel-plugin-istanbul
            // (we will filter the code coverage for non-application files later)
            {
              test: /\.(ts|tsx)$/,
              exclude: /node_modules/,
              use: {
                loader: 'babel-loader',
                options: {
                  presets: ['@babel/preset-typescript'],
                  plugins: [
                    // we could optionally insert this plugin
                    // only if the code coverage flag is on
                    'istanbul',
                  ],
                },
              },
            },
          ],
        },
      },
    },
    experimentalSingleTabRunMode: true,
  },
});

I haven't enclosed a test as it' pretty random - can do at request.

Cypress Version

10.7.0

Node version

16.15.0

Operating System

Linux

Debug Logs

No response

Other

No response

[astone123]

@paulgrant could you provide an example project that uses this configuration with a test so that I can reproduce the issue? That would help us figure out what's going on. Thanks!

[AlexandreGaubert]

I have the same isssue in my github action, I wrote a discussion #24204.

My config is more simpler than @paulgrant :

import { defineConfig } from 'cypress';

export default defineConfig({
  component: {
    viewportHeight: 1080,
    viewportWidth: 1080,
    devServer: {
      framework: 'create-react-app',
      bundler: 'webpack',
    },
  },

  e2e: {
    viewportHeight: 1080,
    viewportWidth: 1920,
    requestTimeout: 30000,
    defaultCommandTimeout: 10000,
    retries: {
      runMode: 2,
      openMode: 0
    }
  },
});

So maybe it's unrelated.

[warrensplayer]

Comment on bottom of https://github.com/cypress-io/cypress/issues/14423 seems related to this. It shows the same stack trace as above.

@AlexandreGaubert Thanks for your comment and write up in the discussion. Are you able to provide a sample project that demonstrates this issue, even if it does not happen all the time or on your development laptops. Here are some tips for providing a Short, Self Contained, Correct, Example. Thanks!

[warrensplayer]

Right now there doesn't seem to be enough information to reproduce the problem on our end. We'll have to close this issue until we can reproduce it. This does not mean that your issue is not happening - it just means that we do not have a path to move forward.

Please open a new issue with a reproducible example and link to this issue. Here are some tips for providing a Short, Self Contained, Correct, Example and our own Troubleshooting Cypress guide.

[paulgrant]

Thanks Alex, one immediate difference I see is in your cypress config file, we are using the component setup, not the e2e setup. https://docs.cypress.io/guides/component-testing/quickstart-react#Configuring-Component-Testing

I believe it is actually down to Azure Pipelines and it not having enough processing power - this is under review for us now. I will review your github over the next day or so - see if I can reproduce the issue! PG On Tuesday, 25 October 2022 at 13:55:47 BST, AlexandreGaubert @.***> wrote:

I've managed to setup a reproducible repository here by forking your examples repository (directory react-cra5-ts).

I've made it pretty close to our configuration, but I didn't reproduced the error in CI. But you can clearly see that tests are reported twice here in the run npm ci job.

But this bug didn't happened in this run here.

So it must be something in this commit

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[paulgrant]

With the latest upgrade of 10.11.0 - we are now seeing extra lines in the error.

A search of the error being displayed would suggest that there is an issue with the teardown of the test with Electron. I was under the impression that nothing explicit needed to be done to team them down. Interestingly - I reduced the tests to just 7 - specPattern: 'src/**/Manage*.spec.{js,jsx,ts,tsx}' And the tests passed.

So it would seem to be a performance issue perhaps - and this occured locally on my development machine - 16GB, 12th gen i7, not in Azure

Resolution macbook-16
Invalid array length
RangeError: Invalid array length
    at Runner.runner._events.pass.<computed> (C:\Users\paulg\AppData\Local\Cypress\Cache\10.11.0\Cypress\resources\app\packages\server\lib\reporter.js:306:11)
    at Runner.emit (node:events:539:35)
    at Runner.emit (node:domain:475:12)
    at Reporter.emit (C:\Users\paulg\AppData\Local\Cypress\Cache\10.11.0\Cypress\resources\app\packages\server\lib\reporter.js:354:46)
    at Object.onMocha (C:\Users\paulg\AppData\Local\Cypress\Cache\10.11.0\Cypress\resources\app\packages\server\lib\project-base.js:273:34)
    at Socket.<anonymous> (C:\Users\paulg\AppData\Local\Cypress\Cache\10.11.0\Cypress\resources\app\packages\server\lib\socket-base.js:259:40)
    at Socket.emit (node:events:527:28)
    at Socket.emit (node:domain:475:12)
    at Socket.emitUntyped (C:\Users\paulg\AppData\Local\Cypress\Cache\10.11.0\Cypress\resources\app\packages\socket\node_modules\socket.io\dist\typed-events.js:69:22)
    at C:\Users\paulg\AppData\Local\Cypress\Cache\10.11.0\Cypress\resources\app\packages\socket\node_modules\socket.io\dist\socket.js:428:39
    at processTicksAndRejections (node:internal/process/task_queues:78:11)

target closed while handling command
Error: target closed while handling command```

[amehta265]

@paulgrant Thank you for the update and stack traces. I was wondering if you could share the tests you are running so that we can try running them on our end?

[zzzorgo]

Faced the issue, apparently the problem is with code on the screenshot. indents counter somehow is negative when we run Array(indents) and it triggers RangeError.

You can work around it by explicitly setting your reporter to spec (don't know why).

import { defineConfig } from 'cypress';

export default defineConfig({
  component: {
    reporter: 'spec',
  },
  e2e: {
    reporter: 'spec',
  },
});

[irving-ventura-shift]

@zzzorgo hero. How did you manage to debug this ? or was it just a guess based on the error message and the line ?

[zzzorgo]

@irving-ventura-shift Yep, just a very informed guess 😌

[amehta265]

@zzzorgo thank you for providing a work around. We will be investigating this further as I don't believe the indents counter should become negative.

@irving-ventura-shift does this potential solution work on your end?

[paulgrant]

@zzzorgo - which version of cypress are you running? This fix did not work for me - Invalid array length RangeError: Invalid array length at indent (C:\Users\paulg\AppData\Local\Cypress\Cache\10.11.0\Cypress\resources\app\node_modules\mocha-7.0.1\lib\reporters\spec.js:45:12)

@amehta265 - putting a null check in the indent function did work : if(isNaN(indents)) { indents = 0; }

So your educated guess would be correct and lies in the indents counter in the cypress version of mocha - 7.0.1 C:\Users\paulg\AppData\Local\Cypress\Cache\10.11.0\Cypress\resources\app\node_modules\mocha-7.0.1\lib\reporters\spec.js

This issue is not addressed in mocha 10.10 https://github.com/mochajs/mocha/blob/master/lib/reporters/spec.js

[zzzorgo]

@paulgrant Cypress version was 10.11.0. You can try using completely different reporter, since the issue is spec specific.

[zack-pronto]

I too have been running into this issue after upgrading to 10.10.0. The runs fail about ~70% of the time immediately after the first test file complete its run:

However, I've been able to execute ~8 consecutive successful suite runs after making the reporter: 'spec' fix that @zzzorgo suggested so that seems to have at least allowed us to start working again without having to downgrade. Thanks @zzzorgo for the help.

[irving-ventura-shift]

@amehta265 I haven't tested it out yet, I was skeptical of it as it didn't feel it would do anything, the code is failing inside the code block dedicated to the spec reporter, the constructor for the reporter is already defaulting to spec, not sure why forcing it to spec once more would solve the issue, but after @zack-pronto report, I will give it a shot and report back.

Ideally I'd love to find a reproduceable case and help debug, but I have not been able to reproduce this locally, it only happens on CI, inconsistently, I'm not running on azure tho...

If @zack-pronto suggestion doesn't works I'll patch cypress on my project and instrument the code a little bit to see why we endup with negative indent values.

However even if it works. I'd still feel better about implementing @paulgrant suggestion, currently the behavior is to throw an exception, with his suggestion you would end up with a potential mal-formatted report.... Which is way less painful than the testing framework randomly throwing exceptions on CI.

[irving-ventura-shift]

@amehta265 pushed changes in my codebase forcing the reporter to spec, still encountering the same issue, any chance we can have a patch with a null check ?

[amehta265]

@irving-ventura-shift the current issue has been routed to the appropriate team at Cypress and it will be prioritized accordingly.

[irving-ventura-shift]

@paulgrant have you found any way to workaround this issue ?

[paulgrant]

@irving-ventura-shift - not presently - I have upgraded to 10.11 and still getting the same issue. I am unfortunatley tied to the spec and the spec format because of the use of SonarCloud.

[irving-ventura-shift]

I'm thinking of adding your changes by using https://www.npmjs.com/package/patch-package I've hundreds of builds a day and plenty of them failing on this issue... first time using that one, I'll let you know how it went if you are interested.

[paulgrant]

I will do the same - looks like I had already installed the package!

[paulgrant]

I couldn't get this to work - the package is not in the cypress node_modules folder of my build:

Invalid array length RangeError: Invalid array length at indent (C:\Users\paulg\AppData\Local\Cypress\Cache\10.11.0\Cypress\resources\app\node_modules\mocha-7.0.1\lib\reporters\spec.js:48:12)

It is in my local appdata folder - which the patch-package doesn't review.

[irving-ventura-shift]

@paulgrant I'm a tiny bit confused, did you get it to work ? in the issue you created in mocha it says that it solved your issue ?

[paulgrant]

@irving-ventura-shift I had a fix that would work locally as I was amending the local appData version of the file - but nothing that would be achieveable in a CI environment.

[irving-ventura-shift]

Well, I have left the boat, I swapped reporters to teamcity 😐

[paulgrant]

I have upgraded to 11.1 and have been able to successfully run it locally, the issue is it is now timing out on Azure Pipelines - so further investigation required.

[distante]

This is also happening to us on Github actions, using reporter=spec lowers the change to get the error, but it still comes

Invalid array length
RangeError: Invalid array length
    at t (<embedded>:1132:173327)
    at R.<anonymous> (<embedded>:1132:173493)
    at R.emit (node:events:539:35)
    at S.emit (<embedded>:2765:16640)
    at Object.onMocha (<embedded>:2812:338633)
    at p.<anonymous> (<embedded>:2812:66203)
    at p.emit (node:events:527:28)
    at p.emitUntyped (<embedded>:2537:41392)
    at <embedded>:2537:48899
    at process.processTicksAndRejections (node:internal/process/task_queues:78:11)

[zack-pronto]

This is still a huge problem for us. This bug ends up killing about 25% of our runs within GitHub Actions. We've tried debugging lots of proposed solutions from the users above and nothing seems to have really helped. We're still getting it very frequently.

Of our last 100 CI test runs, we've had 24 of them randomly fail because of this bug.

That's roughly 1 in 4! When it happens, we have to just restart the run and hope it finishes. Please fix it!

[paulgrant]

An update from me, after upgrading to Cypress 12.5.1, I am still receiving the same issue.

RangeError: Invalid array length at t (<embedded>:2851:153968) at R.<anonymous> (<embedded>:2851:154134) at R.emit (node:events:539:35) at S.emit (<embedded>:4592:16654) at Object.onMocha (<embedded>:4639:425395) at p.<anonymous> (<embedded>:4639:66778) at p.emit (node:events:527:28) at p.emitUntyped (<embedded>:4350:84346) at <embedded>:4350:91863 at process.processTicksAndRejections (node:internal/process/task_queues:78:11) Resolution macbook-16 target closed while handling command Error: target closed while handling command

[paulgrant]

I have finally got this working in 12.6. It was quite a large refactor but the main points:

1. We had a number of nested describe blocks which I think led to the RangeError issue. We were nesting describes to organise them and our Viewport code would also wrap items in a describe.
2. Using the @testing-library/react and using Ken Dodds approach to keep tests clean and simple, I implemented the clean up before every test, not after incase of failure! import { cleanup } from '@testing-library/react'; beforeEach(() => cleanup()); NB* this is the preferred usage of beforeEach as it cleans up state before tests run. https://docs.cypress.io/guides/references/best-practices#Using-after-or-afterEach-hooks
3. I have moved all hook tests out of cypress into jest.

I now have a fully running Cypress 12.7 instance which runs component and e2e tests.

Is it faster than 8? No, not by a long shot. With the trimming and the slimming of tests, best practices followed etc. The tests still timeout on Azure Pipelines. Locally on an i7 12th gen, it runs in a little over 9 minutes.

But that is another story - good luck!