Collect and measure impact of image sizes improvements

[joemcgill]

Purpose

As part of #760, we should define some initial baseline metrics against which we can evaluate the impact of the project and decide how to report them over time.

To start, we have created an initial query of HTTPArchive data that shows the distribution of the sizes error and the the effect on wasted pixels per image caused by selecting the wrong source.

Additional tasks

• [ ] Create a Colab to collect metrics for image sizes improvements.
• [ ] Modify the original HTTPArchive query, to limit it to only images that are the LCP image.
• [ ] Correlate percentage of sites with LCP passing rates with sites with an LCP image with an incorrect sizes attribute resulting in wasted pixels.

[joemcgill]

@felixarntz and @adamsilverstein, now that we've landed auto-sizes for 6.7 (trunk) and have already shipped some initial improvements for enhanced sizes calculation in the Enhanced Responsive Images plugin, let's pick this back up and work on a Collab for monitoring the impact of this work.

Earlier, I started breaking down some "additional tasks", but would love to start by working with you two to better define the objectives for this work. Ideally, I'd like to track both the adoption and effect of auto-sizes, as well as being able to track the adoption and effect of the sizes calculation improvements—both in terms of wasted bytes saved and any potential effect these features have on CWV passing rates.

I don't anticipate auto-sizes to have a big effect on CWV, since it only applies to lazy-loaded images, but would be interesting to confirm that hypothesis. For better sizes calculation, it would be good to see the impact broken into sites using block templates, blocks for content only, and possibly pages not using blocks at all.

Curious what you would add or change from this list, or any other ideas that come to mind.

[adamsilverstein]

One other idea for a metric to look at would be the Lighthouse "Properly Sized Image" audit which we should have available in httparchive (and I don't think is what the current query looks at). Ideally we should see this audit frequency reduced for sites that have better sizes.

[adamsilverstein]

Create a Colab to collect metrics for image sizes improvements.

I have an existing colab started we can expand on. I moved to a gist to make it more accessible: https://gist.github.com/adamsilverstein/6d5111ca510d493559f2289a8fcab428

[felixarntz]

I gathered impact metrics for the accuracy of the sizes attribute from the auto-sizes plugin specifically in this query: https://github.com/GoogleChromeLabs/wpp-research/pull/162

Spreadsheet with the full data

Worth highlighting: At the 70th percentile, we see a reduction of over 40% on desktop and almost 60% on mobile in terms of wasted pixels and wasted bytes.

I also ran a modified version of the query only to get the LCP difference for the similar intersection of sites, and that looks great too, with +8.2% on desktop and +6.2% on mobile (the overall LCP diff in that time for all WordPress sites was only ~+1%, per https://cwvtech.report).

This is all very encouraging so far. 🎉

[joemcgill]

This is encouraging! To repeat my comment on the SQL query:

It's worth noting that this query will include images that do not get auto-sizes applied (e.g. non lazy-loaded images), even when the plugin is added so the impact on affected images might be even greater. Would that require a new custom metric, like image.hasAutoSizes? If so, should we try adding a new custom metric for that prior to the WP 6.7 release?

It's also nice to see a positive correlation between adding this plugin and improved LCP passing rates. There are other possible factors that could have affected the LCP passing rates on those URLs, but this certainly is a positive sign.

[joemcgill]

Now that the auto-sizes feature was released in WP 6.7 last week, we will review the data again once the December CrUX dataset is available.

[adamsilverstein]

Noting that I have updated the data in the colab to the latest available.

[joemcgill]

@adamsilverstein and @felixarntz, with November CWV numbers available this week, can we update the queries we've run to see what the initial impact of adding auto-sizes to 6.7 has been?

[felixarntz]

In https://github.com/GoogleChromeLabs/wpp-research/pull/167, I've gathered data on the reduction of wasted pixels and bytes for all relevant sites that updated to WordPress 6.7, showing the impact at scale, still with great results: On desktop, wasted loaded bytes are over 30% reduced! 🎉

[adamsilverstein]

I also updated the colab with a new section comparing 6.6 & 6.7 (10/1-11/1) sites using the Lighthouse "Properly sized images" audit. We will need to wait for the 12/1 dataset to get more meaningful number for this audit.

[mukeshpanchal27]

@felixarntz Could you please re-run the query and share the results, similar to what you did in https://github.com/WordPress/performance/issues/1186#issuecomment-2392454315?