Assertion of category is not working

[mkathu]

Describe the bug On triggering the scan assertion is not done at the category.

To Reproduce Steps to reproduce the behavior:

1. Configure the page to be scanned
2. Specify the minimum performance score and other metrics for assertion
3. Execute the scan for the page
4. Open the report

Expected behavior Once execution is done assertion should be done should pass/fail based on configured values

Observed: The assertion is not done for category but metrics

Logs/Screenshots

Environment (please complete the following information):

• OS: Ubuntu latest
• Browser : chrome

[volodya315]

I get the same result with assert matrix. For both urls assertions should fail, but everything passes

        assert: {
            assertMatrix: [
                {
                    matchingUrlPattern: "url1",
                    assertions: {
                        'categories:performance': ['error', { minScore: 0.99 }],
                    }
                },
                {
                    matchingUrlPattern: "url2",
                    assertions: {
                        'categories:performance': ['error', { minScore: 0.99 }],
                    }
                }
            ]
        },

It only works if I replace assertMatrix with one assert for every url, but I need to set different categories scores for different environments

      assert: {
            assertions: {
                'categories:performance': ['error', { minScore: 0.9 }],
            },

OS: MacOS 12.0.1 Browser: Chrome

[wac2007]

Some updates on this? I'm having the same problem :/

[svobom57]

I think I'm experiencing the same issue sometimes on my CI. Is it possible it has to do with how Lighthouse runs the assertions?

In the docs there is a section mentioning the aggregation of the results: https://github.com/GoogleChrome/lighthouse-ci/blob/main/docs/configuration.md#aggregation-methods and it seems like the default is optimistic whatever it means. Could this be the issue?