refactor(rome_cli): improve the stability of the `max_diagnostics` tests
Summary
The tests for the --max-diagnostics CLI arguments are failing semi-randomly (it seems to happen mostly with macOS on CI but I managed to reproduce the failure at least once on Windows), this PR tries to improve these tests by counting the console messages emitted for diagnostics but still checking the rest of the console output against a snapshot to try and narrow down the issue
Test Plan
Run the new tests on CI
Deploy Preview for docs-rometools ready!
| Name | Link |
|---|---|
| Latest commit | f80506db4ab6af3b22c29a606e6f97782d1aee63 |
| Latest deploy log | https://app.netlify.com/sites/docs-rometools/deploys/637b7a2e0b0f4e0009e7c8ee |
| Deploy Preview | https://deploy-preview-3729--docs-rometools.netlify.app/playground |
| Preview on mobile | Toggle QR Code...Use your smartphone camera to open QR code link. |
To edit notification comments on pull requests, go to your Netlify site settings.
@leops any chance to get this merged and see if we're able to lower the random failures?
@leops any chance to get this merged and see if we're able to lower the random failures?
Yes we could try to merge it, I'll rebase it and move it out of draft
Comparing refactor(rome_cli): improve the stability of the max_diagnostics tests Snapshot #1 to median since last deploy of rome.tools.
| LCP? | CLS? | TBT? | |
|---|---|---|---|
| Overall Median across all pages and test profiles |
465ms from 274ms |
0.049 from 0.009 |
0ms no change |
| Chrome Desktop Chrome Desktop • Cable |
465ms from 274ms |
0.013 from 0.005 |
88ms from 19ms |
| iPhone, 4G LTE iPhone 12 • 4G LTE |
241ms from 238ms |
0.082 from 0.009 |
0ms no change |
| Motorola Moto G Power, 3G connection Motorola Moto G Power • Regular 3G |
1.2s from 1.07s |
0.049 from 0.009 |
0ms no change |
1 page tested
Home
Browser previews
| Chrome Desktop | iPhone, 4G LTE | Motorola Moto G Power, 3G connection |
|---|---|---|
![]() |
![]() |
![]() |
Most significant changes
![]() |
Value | Budget |
|---|---|---|
| Total JavaScript Size in Bytes Chrome Desktop |
1.33 MB from 86.8 KB |
|
| Total JavaScript Size in Bytes iPhone, 4G LTE |
1.33 MB from 86.8 KB |
|
| Total JavaScript Size in Bytes Motorola Moto G Power, 3G connection |
1.33 MB from 86.8 KB |
|
| Cumulative Layout Shift iPhone, 4G LTE |
0.082 from 0.009 |
|
| Cumulative Layout Shift Motorola Moto G Power, 3G connection |
0.049 from 0.009 |
17 other significant changes: Number of Requests on Chrome Desktop, Number of Requests on iPhone, 4G LTE, Number of Requests on Motorola Moto G Power, 3G connection, Total Blocking Time on Chrome Desktop, Total Page Size in Bytes on Chrome Desktop, Total Page Size in Bytes on iPhone, 4G LTE, Total Page Size in Bytes on Motorola Moto G Power, 3G connection, JS Parse & Compile on Chrome Desktop, Total Image Size in Bytes on Chrome Desktop, Total Image Size in Bytes on iPhone, 4G LTE, Total Image Size in Bytes on Motorola Moto G Power, 3G connection, Total CSS Size in Bytes on Chrome Desktop, Total CSS Size in Bytes on iPhone, 4G LTE, Total CSS Size in Bytes on Motorola Moto G Power, 3G connection, Total HTML Size in Bytes on Chrome Desktop, Total HTML Size in Bytes on iPhone, 4G LTE, Total HTML Size in Bytes on Motorola Moto G Power, 3G connection
Calibre: Site dashboard | View this PR | Edit settings | View documentation



