Specifying `--path` option when generating coverage results in partial/truncated reports.

[alexabreu]

trafficstars

In order to lower our CI costs, we're building once and sharing that build across multiple partition test runs (via ember exam). All seems to work fine, but the lcov.info reports being generated across each test run are both identical and incomplete.

Simply removing the path option, therefore forcing a fresh build prior to running tests, remedies this problem and results in correct coverage reports. This, however, has the unwanted side effect of each test run worker being billed for the added build time.

Did I miss a configuration option to handle this specific use case?

Incomplete lcov.info

TN:
SF:app/formats.js
FNF:0
FNH:0
LF:0
LH:0
BRF:0
BRH:0
end_of_record
TN:
SF:app/resolver.js
FNF:0
FNH:0
LF:0
LH:0
BRF:0
BRH:0
end_of_record
TN:
SF:app/components/esri-icon.js
FNF:0
FNH:0
LF:0
LH:0
BRF:0
BRH:0
end_of_record

Incomplete coverage-summary.json

{"total": {"lines":{"total":0,"covered":0,"skipped":0,"pct":100},"statements":{"total":0,"covered":0,"skipped":0,"pct":100},"functions":{"total":0,"covered":0,"skipped":0,"pct":100},"branches":{"total":0,"covered":0,"skipped":0,"pct":100}}
,"app/formats.js": {"lines":{"total":0,"covered":0,"skipped":0,"pct":0},"functions":{"total":0,"covered":0,"skipped":0,"pct":0},"statements":{"total":0,"covered":0,"skipped":0,"pct":0},"branches":{"total":0,"covered":0,"skipped":0,"pct":0}}
,"app/resolver.js": {"lines":{"total":0,"covered":0,"skipped":0,"pct":0},"functions":{"total":0,"covered":0,"skipped":0,"pct":0},"statements":{"total":0,"covered":0,"skipped":0,"pct":0},"branches":{"total":0,"covered":0,"skipped":0,"pct":0}}
,"app/components/esri-icon.js": {"lines":{"total":0,"covered":0,"skipped":0,"pct":0},"functions":{"total":0,"covered":0,"skipped":0,"pct":0},"statements":{"total":0,"covered":0,"skipped":0,"pct":0},"branches":{"total":0,"covered":0,"skipped":0,"pct":0}}
}

[kategengler]

Can you confirm:

• the build you are sharing was built with the COVERAGE env variable set to true?
• You have set parallel to true in the configuration (https://github.com/kategengler/ember-cli-code-coverage#configuration)
• You are using the coverage-merge command to merge coverage across exam runs

[barryofguilder]

@kategengler We just ran into this in our Ember App and I can confirm that we've done all 3 of those steps. I'm not sure what all information you need, but here's a snippet of the npm scripts we're using in our GitHub Action workflow:

# Builds the app, which all of the exam partions use
# Since this is done in a separate GitHub Action job than ember-exam, I'm not specifying the output-path as
# that's being handle within our action using upload-artifact/download-artifact.
COVERAGE=true ember build --environment=test

#  exam script:
COVERAGE=true ember exam --query=nolint --split=4 --parallel=1

# Called like this in the GitHub Action (done 4 times to cover all partitions):
npm run exam -- --partition=1 --path=dist

# Called in our SonarCloud step
npm run coverage-merge

[kategengler]

@barryofguilder If you follow the same steps locally, does it work? Before you run coverage-merge, you should see several coverage123abc directories.

Could you add a test app here https://github.com/kategengler/ember-cli-code-coverage/tree/master/test-packages with a reproduction?

[barryofguilder]

@kategengler

I was able to reproduce this locally. I do see several coverage_123abc directories (one for each ember-exam run), but they barely contain any coverage info. Way less than what we had in each partition before we added the --path=dist param.

I will try and get a test app uploaded to the repo tomorrow.

[barryofguilder]

I'm still trying to get a test app together to show the issue. I'm struggling to have ember exam partition the tests when there's not many tests. It seems to be running all the tests in each partition.

[ndekeister-us]

@barryofguilder may i ask you what versions of ember-cli-code-coverage + ember-exam are you using? Tried to reproduce your issue with latest versions ([email protected] and [email protected]) in an Ember app having a large test suite (1.5k+ tests) and i'm not able to reproduce, did following steps

// exam script -> COVERAGE=true ember exam --query=nolint --split=4 --parallel=1
COVERAGE=true ember build --environment=test
npm run exam -- --partition=1 --path=dist
npm run exam -- --partition=2 --path=dist
npm run exam -- --partition=3 --path=dist
npm run exam -- --partition=4 --path=dist
npm run coverage-merge

Resulting of having same coverage data as running directly COVERAGE=true ember exam --query=nolint

[barryofguilder]

@ndekeister-us sure, here are the versions we are using:

• ember-exam: "^6.1.0"
• ember-cli-code-coverage: "^1.0.3"
• ember-cli: "~3.25.2"
• ember-source: "3.25.1"

[ndekeister-us]

Thanks, made some try in multiple apps but can't success to reproduce even if we have exact same packages & versions, hope you will success to reproduce it in a test app 🤞

[alexabreu]

Happy to report that the root cause of this bug is due to this TypeScript issue and was indeed resolved by upgrading to v2.0.0.beta-4.

[barryofguilder]

Happy to report that the root cause of this bug is due to this TypeScript issue and was indeed resolved by upgrading to v2.0.0.beta-4.

Ohhh, that would explain why I was never able to reproduce. I wasn't using TypeScript in any of the test apps.

[alexabreu]

So I may have spoken too soon... While it does seem beta-4 improves the situation, we're seeing empty coverage reports for a subset of our partitions. For instance, we'll get 7/10 valid coverage reports and 3 empty files. Strangely, it seems the number of valid reports changes across builds. Does this sound like it's related to #349?

I've tried:

• Eliminating the use of the load-balance flag
• Increasing/decreasing the value used for split flag

What else might I try?

This is what our script calls look like:

coverage.js

module.exports = {
  parallel: true,
}

package.json

...
"test:ci": "ember exam --parallel=2 --load-balance --split=10 --test-port 0",
"posttest:ci": "npx ember coverage-merge",
...

COVERAGE=true npm run test:ci -- --partition=7 --path='test-dist'

[LanceStasinski]

Hi @kategengler, has there been any update regarding this issue? I'm working with the same configuration as @alexabreu and seeing empty coverage reports as described in the previous comment.