lighthouse performance output is significantly different from the browser inbuilt one

[fasatrix]

Hi there, I noticed that the outputted metrics are significantly different from the browser inbuilt one, why is that(am I doing anything wrong)?

test:

import { chromium } from 'playwright';
import type { Browser } from 'playwright';
import { playAudit } from 'playwright-lighthouse';
import { test as base } from '@playwright/test';

export const lighthouseTest = base.extend<
    {},
    { port: number; browser: Browser}
    >({
    port: [
        async ({}, use) => {
            await use(9222);
        },
        { scope: 'worker' },
    ],

    browser: [
        async ({ port }, use) => {
            const browser = await chromium.launch({
                args: [`--remote-debugging-port=${port}`],
                headless: false,
            });
            await use(browser);
        },

        { scope: 'worker' },
    ],

});

lighthouseTest.describe('Lighthouse', () => {
    lighthouseTest('should pass lighthouse tests', async ({ page ,port }) => {
        await page.goto('https://angular.io');
        await playAudit({
            page,
            port,
            opts: { screenEmulation: { disabled: true } },
            thresholds: {
                performance: 40,
                accessibility: 50,
                'best-practices': 50,
                seo: 50,
                pwa: 50,
            },
        });
    });
});

From the inbuilt lighthouse

What I have noticed is that the output from your lib is very closed to the mobile output. However I added opts: { screenEmulation: { disabled: true } } to make sure mobile device is not run

[ryanrosello-og]

hey @fasatrix , I was also experiencing wild numbers when I run my tests. I then stumbled upon this article which has a good explanation as to why the numbers may fluctuate and offers up some suggestions on how to mitigate.

https://github.com/GoogleChrome/lighthouse/blob/master/docs/variability.md

[fasatrix]

hey @fasatrix , I was also experiencing wild numbers when I run my tests. I then stumbled upon this article which has a good explanation as to why the numbers may fluctuate and offers up some suggestions on how to mitigate.

https://github.com/GoogleChrome/lighthouse/blob/master/docs/variability.md

Hey, @ryanrosello-og thanks for that. My tests were run from the same computer on the same network multiple times using the web browser version and this package. So the environment was consistently the same however results were consistently different

[badsyntax]

What I have noticed is that the output from your lib is very closed to the mobile output. However I added opts: { screenEmulation: { disabled: true } } to make sure mobile device is not run

Can you try with different configs?

eg:

import lighthouseDesktopConfig from 'lighthouse/lighthouse-core/config/lr-desktop-config';
// import lighthouseMobileConfig from 'lighthouse/lighthouse-core/config/lr-mobile-config';

await playAudit({
  // ...
  config: lighthouseDesktopConfig,
  // ...
})

[das-en]

I am having the same issue as @fasatrix . Any update on this?

[YonatanKra]

Hi,

I'm having the same thing for a11y as well.

Lighthouse in dev tools reports a11y errors while the a11y score when using playwright-lighthouse is 100.

Here's my config:

	const config = {
		extends: 'lighthouse:default',
		settings: {
			maxWaitForFcp: 15 * 1000,
			maxWaitForLoad: 35 * 1000,
			formFactor: 'desktop',
			screenEmulation: {
				mobile: false,
				width: 1350,
				height: 940,
				deviceScaleFactor: 1,
				disabled: false,
			},
			emulatedUserAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
			skipAudits: [
				'html-has-lang',
				'document-title'
			],
		},
	};

	await testWrapper?.screenshot({
		path: './snapshots/select-a11y.png',
	});

	await playAudit({
		page: page,
		thresholds: {
			accessibility: 100,
		},
		port: 9222,
		reports: {
			formats: {
				html: true,
			},
			name: 'select-a11y',
		},
		config,
	});

In the dev tools I see 13 audits while in the playwright-lighthouse output I see only 3.

Any idea why?

[Kolby-Udacity]

I'm seeing the same problem. Running the test in my browser will return a performance score of ~85, running it in playwright I get ~70, running it in CI I get around ~50.