Audit did not produce a value at all on all URLs after the first on jenkins

[sanderlenaerts]

Describe the bug When using multiple URLs passed through in an array in the lighthouserc.json it only produces values for the first URL from this array.

It does successfully execute the x runs on the other URLs, but it cannot produce a result. This only occurs when running it on Jenkins.

Expected behavior I would expect a result to be generated for all x URLs passed in the lighthouserc.json

Logs/Screenshots 1 result(s) for [1mhttps://myurl.com[0m : 21:05:55
21:05:55 [31m✘[0m [1mcategories[0m.performance failure for [1mminScore[0m assertion 21:05:55 Audit did not produce a value at all. "minScore" might not be a valid assertion for this audit. 21:05:55
21:05:55 expected: >=[32m0.38[0m 21:05:55 found: [31mNaN[0m 21:05:55 [2mall values: NaN, NaN, NaN[0m Environment (please complete the following information):

• Linux Jenkins

[ivandjorgon]

Version @lhci/cli: "^0.8.2"

We are experiencing similar Errors in Bitbucket pipelines. When testing/triggering lighthouse locally (against different environments), it works perfectly fine.

As indicated above, lighthouse executes successfully against 'x' different URLs, but fails with the following Error Messages (I have removed the actual URLs). It specifically fails for "categories.performance failure for minScore assertion":

lhci autorun --collect.settings.chromeFlags='--no-sandbox' ✅ .lighthouseci/ directory writable ✅ Configuration file found ✅ Chrome installation found Healthcheck passed!

Running Lighthouse 3 time(s) on .... Run 1...done. Run 2...done. Run 3...done. ....

Done running Lighthouse!

Checking assertions against 4 URL(s), 12 total run(s) 1 result(s) for .... : ✘ categories.performance failure for minScore assertion Audit did not produce a value at all. "minScore" might not be a valid assertion for this audit. expected: >=0.6 found: NaN all values: NaN, NaN, NaN 1 result(s) for .... : ✘ categories.performance failure for minScore assertion Audit did not produce a value at all. "minScore" might not be a valid assertion for this audit. expected: >=0.6 found: NaN all values: NaN, NaN, NaN 1 result(s) for .... : ✘ categories.performance failure for minScore assertion Audit did not produce a value at all. "minScore" might not be a valid assertion for this audit. expected: >=0.6 found: NaN all values: NaN, NaN, NaN 1 result(s) for .... : ✘ categories.performance failure for minScore assertion Audit did not produce a value at all. "minScore" might not be a valid assertion for this audit. expected: >=0.6 found: NaN all values: NaN, NaN, NaN Assertion failed. Exiting with status code 1.

Note that assertion for categories.performance is defined as: 'categories:performance': ['error', { minScore: 0.6, aggregationMethod: 'median-run' },]

The assertions for categories.accessibility, categories.best-practices and categories-seo are defined somewhat differently - example below: 'categories:accessibility': ['error', { minScore: 0.75, aggregationMethod: 'median' },]

(At the moment, we are intentionally testing with low assertion levels and trying out different things.)

[ivandjorgon]

Workaround: Remove categories.performance assertion altogether -> NaN values/errors will no longer appear, and the pipeline is successfully passing.

[algarciatech]

Any solutión for this problem?, I have this problem and NaN response in assertion for categories.performance