tools icon indicating copy to clipboard operation
tools copied to clipboard

refactor(rome_cli): improve the stability of the `max_diagnostics` tests

Open leops opened this issue 3 years ago • 1 comments

Summary

The tests for the --max-diagnostics CLI arguments are failing semi-randomly (it seems to happen mostly with macOS on CI but I managed to reproduce the failure at least once on Windows), this PR tries to improve these tests by counting the console messages emitted for diagnostics but still checking the rest of the console output against a snapshot to try and narrow down the issue

Test Plan

Run the new tests on CI

leops avatar Nov 14 '22 16:11 leops

Deploy Preview for docs-rometools ready!

Name Link
Latest commit f80506db4ab6af3b22c29a606e6f97782d1aee63
Latest deploy log https://app.netlify.com/sites/docs-rometools/deploys/637b7a2e0b0f4e0009e7c8ee
Deploy Preview https://deploy-preview-3729--docs-rometools.netlify.app/playground
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify[bot] avatar Nov 14 '22 16:11 netlify[bot]

@leops any chance to get this merged and see if we're able to lower the random failures?

ematipico avatar Nov 21 '22 12:11 ematipico

@leops any chance to get this merged and see if we're able to lower the random failures?

Yes we could try to merge it, I'll rebase it and move it out of draft

leops avatar Nov 21 '22 13:11 leops

Comparing refactor(rome_cli): improve the stability of the max_diagnostics tests Snapshot #1 to median since last deploy of rome.tools.

LCP? CLS? TBT?
Overall
Median across all pages and test profiles
465ms
from 274ms
0.049
from 0.009
0ms
no change
Chrome Desktop
Chrome Desktop • Cable
465ms
from 274ms
0.013
from 0.005
88ms
from 19ms
iPhone, 4G LTE
iPhone 12 • 4G LTE
241ms
from 238ms
0.082
from 0.009
0ms
no change
Motorola Moto G Power, 3G connection
Motorola Moto G Power • Regular 3G
1.2s
from 1.07s
0.049
from 0.009
0ms
no change

1 page tested

 Home

Browser previews

Chrome Desktop iPhone, 4G LTE Motorola Moto G Power, 3G connection
Chrome Desktop iPhone, 4G LTE Motorola Moto G Power, 3G connection

Most significant changes

Value Budget
Total JavaScript Size in Bytes
Chrome Desktop
1.33 MB
from 86.8 KB
Total JavaScript Size in Bytes
iPhone, 4G LTE
1.33 MB
from 86.8 KB
Total JavaScript Size in Bytes
Motorola Moto G Power, 3G connection
1.33 MB
from 86.8 KB
Cumulative Layout Shift
iPhone, 4G LTE
0.082
from 0.009
Cumulative Layout Shift
Motorola Moto G Power, 3G connection
0.049
from 0.009

17 other significant changes: Number of Requests on Chrome Desktop, Number of Requests on iPhone, 4G LTE, Number of Requests on Motorola Moto G Power, 3G connection, Total Blocking Time on Chrome Desktop, Total Page Size in Bytes on Chrome Desktop, Total Page Size in Bytes on iPhone, 4G LTE, Total Page Size in Bytes on Motorola Moto G Power, 3G connection, JS Parse & Compile on Chrome Desktop, Total Image Size in Bytes on Chrome Desktop, Total Image Size in Bytes on iPhone, 4G LTE, Total Image Size in Bytes on Motorola Moto G Power, 3G connection, Total CSS Size in Bytes on Chrome Desktop, Total CSS Size in Bytes on iPhone, 4G LTE, Total CSS Size in Bytes on Motorola Moto G Power, 3G connection, Total HTML Size in Bytes on Chrome Desktop, Total HTML Size in Bytes on iPhone, 4G LTE, Total HTML Size in Bytes on Motorola Moto G Power, 3G connection

Calibre: Site dashboard | View this PR | Edit settings | View documentation

calibre-analytics[bot] avatar Nov 21 '22 13:11 calibre-analytics[bot]