Question about: Some test suites likely terminated without passing on results

[aram-bigid]

Hi,

First off, thanks for the repo and work :)

So i want to use this package in our tests, but am seeing the following error randomly on my test runs: Some test suites likely terminated without passing on results

I know that it's a validation, but do we know why this is happening?

Thanks, -arnon

[mahanthi-arun-kumar]

I am also facing a similar kind of issue, Found test suites does not match results. Test suites found: 1036 Test suite results: 44 Some test suites likely terminated without passing on results, exit with error

This is the error I am getting: npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! [email protected] cypress:parallel: cypress-parallel -s cypress:run -t 2 -d './src' -a '"--config baseUrl=http://localhost:3000 integrationFolder=src testFiles=**/integration-test/*cypress.js" '` npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the [email protected] cypress:parallel script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in: npm ERR! /Users/maarunkumar/.npm/_logs/2022-03-26T05_14_05_325Z-debug.log`

[ronit15]

I am also facing same issue any updates?

[Oryhara]

Mahanthi-arun-kumar, I believe your problem is in the command line for cypress-parallel.
"-d './src'" tells cypress-parallel to look in your src directory for cypress test suite files. cypress-parallel treats every file it finds as a test suite file. Hence "Test suites found: 1036" that is every code file in your src directory. in your -a arguments passed to cypress itself, you declare testFiles=**/integration-test/*cypress.js" which tells cypress where to find test suite files. further, cypress can distinguish between files that contain tests and files which do not. This accounts for Test suite results: 44 You likely only have 44 test suites. But cypress-parallel thinks you have 1036 test suites, and so expects 1036 test suite results. Upon getting less than 1036 results, it throws an error. If you only pass directories which contain only cypress test suite files to cypress-parallel, that should fix your problem. I ran into this same issue and that fixed it for me.

[olga-govor]

I passed correct tests directory with correct regexp and in 95% tests pass without any issues. But sometimes randomly I've got results like Found test suites does not match results. Test suites found: 71 Test suite results: 61 Some test suites likely terminated without passing on results, exit with error

It mentions specific suites that were not found, but I see results for these suites. Any ideas why it could happen?

[Densf2]

Also tests failed with the same message in console ^^ - if launch by specific tag.

[chojnicki]

@tnicola is this package abandoned? Maybe someone has working fork?

This script should using glob/pattern for spec files, there is no reason to do this differently than Cypress itself. I'm using Cypress for component testing, and like many other people I have spec files together with component files. All my specs are passing, but I'm still getting false error from cypress-parallel :/

[SaniaSid]

Any update on this one ? Facing this issue and there is no information on why this is happening

[prashantpundhir]

I am also facing similar issue, but for me, results shows only 0 even though tests are running successfully. Any updates on this ticket?

[ezunigaf90]

Has anyone managed to solve/bypass this issue?

Still getting this issue on version 0.13.0

[olga-govor]

For us it was an issue with different threads that used the same results folder. After small update to put results in different folders for different threads (unfortunately it's hardcoded in source lib) we didn't meet these problem. For now 2nd month without this issue

[ezunigaf90]

Since we don't get answers here I've been digging through the code because I think this plugin is really nice and I think I found what the issue was for us...

First of all, the error I was getting is:

Found test suites does not match results. 
Test suites found: 68 
Test suite results: 0 
Some test suites likely terminated without passing on results, exit with error
The following tests suites are missing results: { test1, test2, ... all 68 files }

So, it was false failing even if all the tests passed and the statement triggering the error is this: https://github.com/tnicola/cypress-parallel/blob/710dd191692c8f11c2823183fcfa36fa81fcd91e/lib/cli.js#L99

To make shorter the story, after debugging it I realized that for "some" unknown reason none of the tests was generating the .json file needed and created automatically by this plugin in order to output the results, it also creates a runner-results folder for that but basically it was empty after any execution and that's why Test suite results: 0 (timeMap.size). I continued debugging and found where this issue started to happen:

We have mochawesome, mocha-junit-reporter and cypress-multi-reporters, so our code looked like this:

cypress v10.11 mochawesome v6.2.2 cypress-multi-reporters v1.6.1 mocha-junit-reporter v2.0.2 cypress-parallel v0.13.0

Piece of our cm-desktop.config.js file
module.exports = defineConfig({
	e2e: {
	        specPattern: 'cypress/user/*.test.js',
		reporter: 'cypress-multi-reporters',
		reporterOptions: {
			configFile: 'cypress/config/user/reporter-config.json',
		}
	}
});

Our reporter-config.json file
{
  "reporterEnabled":"mochawesome, mocha-junit-reporter",
  "mochaJunitReporterReporterOptions":{
    "mochaFile":"cypress/results/user/dsk/user-[hash].xml"
  },
  "mochawesomeReporterOptions":{
    "reportDir":"cypress/results/user/dsk",
    "overwrite":false,
    "html":false,
    "json":true
  }
}

Our package.json scripts
"cm:dsk:user": "cypress run --config-file ./cypress/config/user/cm-desktop.config.js --browser chrome",
"cm:run:dsk:user:parallel": "cypress-parallel -s cm:dsk:user -t 4 -d ./cypress/user",

^ That way, I was getting the error... So, after almost a full day of trial and error, I found that when I was executing this way then the multi-reporter-config.json file generated by default by the plugin was writing this lines:

{
  "reporterEnabled": "cypress-parallel/json-stream.reporter.js, cypress-parallel/simple-spec.reporter.js"
}

So, it immediately made me think that my reporter and reporterOptions flag was not being read... why? I don't know the answer, I thought it should have worked but looks like it's not working in that way, maybe a bug?? anyway... this statement confirmed my theory: https://github.com/tnicola/cypress-parallel/blob/710dd191692c8f11c2823183fcfa36fa81fcd91e/lib/thread.js#L36 effectively it was not recognizing my reporting flags.

Note: I tried passing -r and -o and a lot of ways/combinations but can't make it work.

The solution? ... I made it work by doing this:

1. Removed the reporter and reporterOptions flags from the cm-desktop.config.js file (yes, deleted those lines from there)
2. Our reporter-config.json file now looks like this:

{
  "reporterEnabled":"mochawesome, cypress-parallel/json-stream.reporter.js, cypress-parallel/simple-spec.reporter.js, mocha-junit-reporter",
    "mochawesomeReporterOptions":{
      "reportDir":"cypress/results/user/dsk",
      "overwrite":false,
      "html":false,
      "json":true
    },
  "cypressParallelSimpleSpecReporterJsReporterOptions": {
    "reportDir": "runner-results"
  },
  "mochaJunitReporterReporterOptions":{
    "mochaFile":"cypress/results/user/dsk/user-[hash].xml"
  }
}

3. Our package.json now look like this:

"cm:dsk:user": "cypress run --config-file ./cypress/config/user/cm-desktop.config.js --browser chrome",
"cm:run:dsk:user:parallel": "cypress-parallel -s cm:dsk:user -t 4 -d ./cypress/user -p 'cypress/config/user/reporter-config.json'",

I found that -p flag in the settings file: https://github.com/tnicola/cypress-parallel/blob/710dd191692c8f11c2823183fcfa36fa81fcd91e/lib/settings.js#L50 which is not documented in README but it was the correct for us, now I'm not passing a reporter flag and letting the reporter-config.json to drive the enabled reporters.

Hope it helps anyone to solve the issue, I don't know if it's a bug or a miss by me while following instructions but that's what it is.

cc @tnicola

[guychienll]

hi @ezunigaf90 May I know after your change above , has this issue happened again ?

[guychienll]

hi @olga-govor May I know you said hardcoded part is to change runner-results/ folder path ? thanks !

[olga-govor]

hi @guychienll in lib/cli.js you can find such line of code const resultsPath = path.join(process.cwd(), 'runner-results'); So here the name of directory for results is hardcoded. I've added here parameter that could be added from command line, so now if I use this plugin for different types of tests run in one workspace it wrote results in different directories. After this we don't have problems with empty or not full results

So changes in settings.js

• add option for CMD .option('runnerResults', { alias: 'x', type: 'string', description: 'Path where cypress results will be located' }
• add read and default value for it runnerResults: argv.runnerResults ? argv.runnerResults : 'runner-default',

in cli.js: const resultsPath = path.join(process.cwd(), settings.runnerResults);

in shared-config.js: const resultsPath = path.join(process.cwd(), ${settings.runnerResults});

[guychienll]

hi @olga-govor thanks lots ! you are awesome !

[guychienll]

hi @olga-govor thank your reply , But I want to know how do you wrote the result in diff folder ? If it's what you mean (the PR below has implementation), How to trigger the command once and wrote results in different folders at the same time? https://github.com/tnicola/cypress-parallel/pull/176

[kganczarek]

Hello, any news on this? Im wondering how this can be fixed, it is a little annoying that same build is failing due to missing results. I'm running one testing procedure with multiple test suits split into 4 threads. Maybe there is any workaround?

[mattweberio]

The issue "Some test suites likely terminated without passing on results" can happen for valid reasons, such as the reason in the error message. However, this is a bug and essentially makes this library unusable without forking. We can verify that the test suite output results and the JSON file exist, but in the mocha folder, there is no JSON for it. As mentioned by another poster above, the tests are overwritten and they had to modify the source to use different folders. It would be nice to get a bug fix, but from what I gather, we're on our own if we decide to use this library.

[igorperic17]

This is too irritating, I would rather let the tests run longer than fail randomly.

Unfortunately, this was a deal breaker for us at https://coretex.ai so we are moving away from using cypress-parallel until this issue is fixed.