cockpit icon indicating copy to clipboard operation
cockpit copied to clipboard

tests: change reference-image to fedora-42

Open tomasmatus opened this issue 6 months ago • 6 comments

tomasmatus avatar May 28 '25 10:05 tomasmatus

somehow this anton guy now logs in

https://cockpit-logs.us-east-1.linodeobjects.com/pull-22053-d1ec974d-20250528-115659-fedora-42/log.html

tomasmatus avatar May 28 '25 13:05 tomasmatus

I can only see that anton could possibly log in in the testRootLogin test. Probably the easiest solution is to just mock this date as it doesn't affect the test in any way.

tomasmatus avatar May 29 '25 10:05 tomasmatus

bleh why do builder and anton users suddenly appear with lastlog info https://cockpit-logs.us-east-1.linodeobjects.com/pull-22053-7eb81774-20250529-110520-fedora-42/pixeldiff.html#TestAccounts-testBasic-users-page-mobile

it probably only fails on mobile because the date does not fit on a single line and other things get pushed around as a result

tomasmatus avatar May 29 '25 12:05 tomasmatus

let's retry this with the new image

tomasmatus avatar May 30 '25 07:05 tomasmatus

The changed pixels should have regressed by now in this PR.

jelly avatar Jun 17 '25 07:06 jelly

Problem might indeed be wtmp/lastlog not getting cleared but we are clearly doing this wrong... We should clear it in the test which logins as that user instead of randomly in tests where pixel tests / expected state fails such as:

        # Clean out the relevant logfiles
        m.execute("truncate -s0 /var/log/{[bw]tmp,lastlog} /run/utmp")
        m.execute("rm -f /var/lib/lastlog/lastlog2.db /var/lib/wtmpdb/wtmp.db")

jelly avatar Jun 23 '25 15:06 jelly

The fixes and pixel ref updates LGTM so far. However, there is a lot of pixel noise on rounded corners -- what is the situation here? Do we need to accept these new ones, or will these be equally brittle -- i.e. if we land this, will this make the flakiness worse?

However, these three sizes are different:

  • https://cockpit-logs.us-east-1.linodeobjects.com/pull-22053-4a1f0a24-20250627-090440-fedora-42-other/pixeldiff.html#TestHistoryMetrics-testEvents-metrics-history-compressed-hour
  • https://cockpit-logs.us-east-1.linodeobjects.com/pull-22053-4a1f0a24-20250627-090440-fedora-42-other/pixeldiff.html#TestHistoryMetrics-testEvents-metrics-history-compressed-hour-medium
  • https://cockpit-logs.us-east-1.linodeobjects.com/pull-22053-4a1f0a24-20250627-090440-fedora-42-other/pixeldiff.html#TestHistoryMetrics-testEvents-metrics-history-compressed-hour-dark

There the expander is hovered on, which smells like "real mouse clicks". I'm not sure how this would be related to updating to F42 though -- is that also a known flake?

martinpitt avatar Jun 27 '25 09:06 martinpitt

The fixes and pixel ref updates LGTM so far. However, there is a lot of pixel noise on rounded corners -- what is the situation here? Do we need to accept these new ones, or will these be equally brittle -- i.e. if we land this, will this make the flakiness worse?

Hard to say if it goes any worse then our current 60% flake

However, these three sizes are different:

* https://cockpit-logs.us-east-1.linodeobjects.com/pull-22053-4a1f0a24-20250627-090440-fedora-42-other/pixeldiff.html#TestHistoryMetrics-testEvents-metrics-history-compressed-hour

* https://cockpit-logs.us-east-1.linodeobjects.com/pull-22053-4a1f0a24-20250627-090440-fedora-42-other/pixeldiff.html#TestHistoryMetrics-testEvents-metrics-history-compressed-hour-medium

* https://cockpit-logs.us-east-1.linodeobjects.com/pull-22053-4a1f0a24-20250627-090440-fedora-42-other/pixeldiff.html#TestHistoryMetrics-testEvents-metrics-history-compressed-hour-dark

There the expander is hovered on, which smells like "real mouse clicks". I'm not sure how this would be related to updating to F42 though -- is that also a known flake?

I have seen this flake before it is obviously an issue do we just ignore that part?

jelly avatar Jun 27 '25 09:06 jelly

I don't think we should entirely ignore the expander. It's part of the pixel test. In #22113 I am working on a hook to make it possible to avoid this. I'll get back to that next week.

martinpitt avatar Jun 27 '25 10:06 martinpitt

#22113 also fixes the hover expander part of the metrics flakes. I'd say, let's land that, rebase to fix the pixel flake, lean on the retry key until this goes green, and then fix these other damn metrics flakes?

martinpitt avatar Jul 01 '25 07:07 martinpitt

networking pixel changes are expected and correct. expensive and storage are green.

other TestKeys-testPrivateKeys-ssh-keys-dialog-rtl is once again some annoying pixel noise which really really isn't distro specific. Let's rather retry that at least once to compare. TestSystemInfo.testBasic pixels are expected (distro version).

martinpitt avatar Jul 07 '25 13:07 martinpitt

Pushed the sane new pixels.

martinpitt avatar Jul 07 '25 14:07 martinpitt