Many new assertion errors giving score 0

[villelahdenvuo]

Describe the bug I haven't changed my assertions and I haven't made any drastic changes to our code and suddenly I get many assertion failures in my tests even though according to the report they are okay.

To Reproduce

1. Run Lighthouse with preset: lighthouse:no-pwa
2. npx --yes --package=@lhci/cli@latest -- lhci autorun

Expected behavior Passed asserts

Logs/Screenshots Note: Unused javascript has been changed from the preset and it used to pass. This is the report generated which shows the assets should pass. Here's the exact same config/code running on 0.12:

Environment (please complete the following information):

• OS: Linux
• Browser Chrome
• Version 11.4.0

[chrismcmacken]

Same issue in my project

[connorjclark]

Lighthouse 11.2 changed how many audits are scored to be based directly on the estimated metric savings. These audits are an example of that.

https://github.com/GoogleChrome/lighthouse/pull/15447 https://github.com/GoogleChrome/lighthouse/pull/15447/files#diff-e65b7054d7252727360d6dfd95b8d36f87da23cd3f9b28a794aaf5d3720e866fR312 etc.

It was not considered how this change would effect LHCI. We'll need to discuss internally how to address this. Given the time of year, this likely won't be solved until after the new year. In the meantime, a few options for you:

• you can update your assertions to use something other than score, like numericValue
• stick to the previous LHCI for now
• force LHCI to use Lighthouse 11.1 or prior

[villelahdenvuo]

Thanks for the details! I'll stick to 0.12.0 for now and keep an eye on this issue to see how to move forward.

[xavierfoucrier]

Same on my side! Thanks for highlighting this in a dedicated issue.

Waiting for the next version @connorjclark 😎

[davidlj95]

Just in case someone's looking for a workaround for now for total-byte-weight using a numeric value as mentioned above:

{
  "ci": {
    "assert": {
      "assertions": {
        "total-byte-weight": ["error", {"maxNumericValue": 1677721600}]
      }
    }
  }
}

This ensures total byte size of requested resources stays under 1,600 KiB as docs recommend

[villelahdenvuo]

This workaround works for us, maybe it's sufficient to document that you can change the asserts like this if needed?

[nstringham]

It seems to me like this is fixed now

[davidlj95]

Yup, looks fixed for me 2 in 0.14