Improved assertMatrix overlap behavior

[ptmkenny]

I was trying to use assertMatrix with preset like this:

myconfig.yml

  assert:
    preset: "lighthouse:recommended"
    assertMatrix:
      - matchingUrlPattern: "devserver.com"
        assertions:
          is-crawlable:
            - off # Dev servers are not crawlable.

However, this results in an error:

Error: Cannot use assertMatrix with other options
    at getAllAssertionResults (/usr/local/lib/node_modules/@lhci/cli/node_modules/@lhci/utils/src/assertions.js:486:13)
    at Object.runCommand (/usr/local/lib/node_modules/@lhci/cli/src/assert/assert.js:58:22)
    at run (/usr/local/lib/node_modules/@lhci/cli/src/cli.js:106:23)
    at Object.<anonymous> (/usr/local/lib/node_modules/@lhci/cli/src/cli.js:137:1)
    at Module._compile (internal/modules/cjs/loader.js:1063:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
    at Module.load (internal/modules/cjs/loader.js:928:32)
    at Function.Module._load (internal/modules/cjs/loader.js:769:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
    at internal/main/run_main_module.js:17:47

What I'm trying to do: I want to use the default checks, but I have a few specific pages that I know will fail a few specific checks. So I was hoping to use assertMatrix to disable such checks for those specific pages.

If I remove preset, I can use assertMatrix, but then I have to create the entire list of assertions that "lighthouse-recommended" checks for myself. In that case, it would be easier to create several different config files for each case that needs special exceptions... but I was hoping to avoid creating multiple config files by using assertMatrix. Is there some way to do that?

[patrickhulce]

Thanks for filing @ptmkenny!

Does this work for you?

 assert:
    assertMatrix:
      - matchingUrlPattern: "devserver.com"
        preset: "lighthouse:recommended"
        assertions:
          is-crawlable:
            - off # Dev servers are not crawlable.

[ptmkenny]

Thank you, that allowed me to get around the Error: Cannot use assertMatrix with other options.

However, now I have a different issue. When I run lighthouse, I'm still getting the crawl error even though I have turned it off.

Here's the actual structure of myconfig.yml:

  assert:
    assertMatrix:
      - matchingUrlPattern: ".*" # Every page.
        preset: "lighthouse:recommended"
        assertions:
          no-unload-listeners: 'off' # New Relic screws this up.
          maskable-icon: 'off' # https://github.com/RealFaviconGenerator/realfavicongenerator/issues/415
      - matchingUrlPattern: "devserver.com"
        assertions:
          is-crawlable:
            - off # Dev servers are not crawlable.

I thought that the issue might be caused by some kind of mistake in the matchingUrlPattern for devserver.com, so I tried this:

  assert:
    assertMatrix:
      - matchingUrlPattern: ".*" # Every page.
        preset: "lighthouse:recommended"
        assertions:
          no-unload-listeners: 'off' # New Relic screws this up.
          maskable-icon: 'off' # https://github.com/RealFaviconGenerator/realfavicongenerator/issues/415
      - matchingUrlPattern: ".*"
        assertions:
          is-crawlable:
            - off # Dev servers are not crawlable.

But in this case too, I still see the is-crawlable error.

[patrickhulce]

Ah, I see. The entries in assertMatrix don't override each other. They're additive. assertMatrix + matchingUrlPattern is like saying "for all URLs that match this pattern, always assert at least these things". You'd need to configure .* to be the minimal set of things that applies to all pages, and then add in individual URLs.

I see two main features that would help make this situation easier.

1. excludingUrlPattern which would be the inverse of matchingUrlPattern (meaning you flip is-crawlable to 'on' and matchingUrlPattern to excludingUrlPattern in your 2nd example, add is-crawlable: 'off' to all pages to achieve the desired effect)
2. assertMatrixOverlapBehavior which would control whether to use additive style today, merge which would basically do an Object.assign (based on your config I'm assuming this is what you thought would happen?), or replace which would only use the first matching pattern.

Do either of those sound especially useful for your situation?

[ptmkenny]

I think in my case the second option would be more helpful. An even more complete example of what I'm trying to do:

  assert:
    assertMatrix:
      - matchingUrlPattern: ".*" # Every page.
        preset: "lighthouse:recommended"
        assertions:
          no-unload-listeners: 'off' # New Relic screws this up.
          maskable-icon: 'off' # https://github.com/RealFaviconGenerator/realfavicongenerator/issues/415
      - matchingUrlPattern: "devserver.com"
        assertions:
          is-crawlable:
            - off # Dev servers are not crawlable.
      - matchingUrlPattern: "user" # /user path
        assertions:
          meta-description:
            - off # User pages do not have a meta description.
      - matchingUrlPattern: "books" # Book pages have amazon links.
        assertions:
          use-text-compression:
            - off # Cannot customize Amazon-injected code.

So ideally, I want to test all of the pages with lighthouse:recommended, but I need to exempt several different paths from various checks because of things I have no control over (third-party code, issues with the CMS I cannot easily change, etc.) I was trying to find a way to define a set of default, very strict criteria and then make a list of exceptions from certain tests for specific paths.

[patrickhulce]

Got it 👍

I was trying to find a way to define a set of default, very strict criteria and then make a list of exceptions from certain tests for specific paths.

yeah we would love to encourage this :)

[klarkc]

Check this workaround: https://github.com/GoogleChrome/lighthouse-ci/issues/208#issuecomment-784501105

[RyanClementsHax]

Is this something that the team is still interested in looking into?