Intermittent UI test failure - < StartupExcessiveResourceUseTest. verifyRunBlockingAndStrictModeSuppresionCount >

[AndiAJ]

Firebase Test Run: Firebase link

Stacktrace:

java.lang.AssertionError: ConstraintLayout being a common direct descendant of a RecyclerView count does not match expected count.

If this PR removed a ConstraintLayout being a common direct descendant of a RecyclerView call, great! Please decrease the count.

Did this PR add or call code that increases the ConstraintLayout being a common direct descendant of a RecyclerView count?
Did you know that ConstraintLayouts are slow to inflate and are primarily used to flatten deep view hierarchies so can be under-performant as a common RecyclerView child?
Please do your best to implement a solution without adding ConstraintLayout being a common direct descendant of a RecyclerView calls.
Please consult the perf team if you have questions or believe that having this call
is the optimal solution.

Please note that we're not sure if this is a useful metric to assert: with your feedback, we'll find out over time if it is or is not. expected:<4> but was:<5> at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.failNotEquals(Assert.java:835) at org.junit.Assert.assertEquals(Assert.java:120) at org.mozilla.fenix.perf.StartupExcessiveResourceUseTest.verifyRunBlockingAndStrictModeSuppresionCount(StartupExcessiveResourceUseTest.kt:90)

Build: 5/6 Main

cc @mcomella @MarcLeclair

┆Issue is synchronized with this Jira Story

[AndiAJ]

Failed 2x on #25133

[AndiAJ]

Failed 1x on #25105 Firebase link Firebase link

[AndiAJ]

Failed 1x on #25143 Firebase link Firebase link

[Amejia481]

An overview of why the test is failing: The test have multiple performance metrics on the home screen, the one that it's failing was counting the specific number of items that we had on the screen, As the messaging framework now adds an additional item the test fails. Before we had 4 items now we have 5 with the default browser message card, this is what we are addressing on the PR. As the messaging framework loads the items async we have to use a range, as sometimes we could have 4 items or 5 items on the screen.

[AndiAJ]

Thanks @Amejia481 for the fix ! 🙏🏽

[Amejia481]

Thanks to you for all the help and sorry for the inconveniences :)

[AndiAJ]

Flaky 1x on 5/10 Main Firebase link

java.lang.AssertionError: ConstraintLayout being a common direct descendant of a RecyclerView count does not match expected count.

If this PR removed a ConstraintLayout being a common direct descendant of a RecyclerView call, great! Please decrease the count.

Did this PR add or call code that increases the ConstraintLayout being a common direct descendant of a RecyclerView count?
Did you know that ConstraintLayouts are slow to inflate and are primarily used to flatten deep view hierarchies so can be under-performant as a common RecyclerView child?
Please do your best to implement a solution without adding ConstraintLayout being a common direct descendant of a RecyclerView calls.
Please consult the perf team if you have questions or believe that having this call
is the optimal solution.

Please note that we're not sure if this is a useful metric to assert: with your feedback, we'll find out over time if it is or is not.actual: 2 at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.assertTrue(Assert.java:42) at org.mozilla.fenix.perf.StartupExcessiveResourceUseTest.verifyRunBlockingAndStrictModeSuppresionCount(StartupExcessiveResourceUseTest.kt:92)

[AndiAJ]

Failed 1x on 5/13 Main @Amejia481 Firebase link Firebase link

java.lang.AssertionError: ConstraintLayout being a common direct descendant of a RecyclerView count does not match expected count.

If this PR removed a ConstraintLayout being a common direct descendant of a RecyclerView call, great! Please decrease the count.

Did this PR add or call code that increases the ConstraintLayout being a common direct descendant of a RecyclerView count?
Did you know that ConstraintLayouts are slow to inflate and are primarily used to flatten deep view hierarchies so can be under-performant as a common RecyclerView child?
Please do your best to implement a solution without adding ConstraintLayout being a common direct descendant of a RecyclerView calls.
Please consult the perf team if you have questions or believe that having this call
is the optimal solution.

Please note that we're not sure if this is a useful metric to assert: with your feedback, we'll find out over time if it is or is not.actual: 3 at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.assertTrue(Assert.java:42) at org.mozilla.fenix.perf.StartupExcessiveResourceUseTest.verifyRunBlockingAndStrictModeSuppresionCount(StartupExcessiveResourceUseTest.kt:92)

[Amejia481]

👋🏽 @AndiAJ I'll take a look!

[Amejia481]

@AndiAJ the performance team could help to answer the usefulness of the assertion, as they are the code owner cc @mcomella. In my opinion, it looks like the fakyness of the assertion has increase overtime, as it looks like the UI components on the home screen are more async (some times they could be there or not) but the expected count is a fixed number.

[mcomella]

as they are the code owner cc

We want to move towards a self service model for this test (rather than the performance team as the code owner) so feel free to remove checks that you feel aren't helpful and of course we can help you understand if that makes sense or not.

I opened a (not yet reviewed) PR to clarify what this test is and what this heuristic is intended to represent: https://github.com/mozilla-mobile/fenix/pull/25320 Please comment on that PR if there's something I can add that would have helped you better understand the current failure!

it looks like the fakyness of the assertion has increase overtime, as it looks like the UI components on the home screen are more async (some times they could be there or not) but the expected count is a fixed number.

If we're seeing a variable number of views, some likely causes are:

• the test no longer waits until start up has completed before it returns so that it measures a variable number of views. If this is the case, that's bad and we should fix it – the test might let performance regressions get into the product
• I've previously seen the home screen unexpectedly inflate views that are off screen because that data returns before the data of views higher in the screen. If that happens, the number of views inflated would be variable

That being said, the RecyclerView -> ConstraintLayout children detector is an experimental metric that we're not 100% certain will help prevent performance regressions. You can remove the heuristic/check if you don't think it's going to catch performance regressions or you don't think fixing the intermittent issue is worth the time. Please let me know if you have more questions! fwiw, if you have further questions, maybe it'd be easier to resolve on Zoom.

[AndiAJ]

Flaky 1x on 5/24 Main Firebase link

java.lang.AssertionError: ConstraintLayout being a common direct descendant of a RecyclerView count does not match expected count.

If this PR removed a ConstraintLayout being a common direct descendant of a RecyclerView call, great! Please decrease the count.

Did this PR add or call code that increases the ConstraintLayout being a common direct descendant of a RecyclerView count?
Did you know that ConstraintLayouts are slow to inflate and are primarily used to flatten deep view hierarchies so can be under-performant as a common RecyclerView child?
Please do your best to implement a solution without adding ConstraintLayout being a common direct descendant of a RecyclerView calls.
Please consult the perf team if you have questions or believe that having this call
is the optimal solution.

Please note that we're not sure if this is a useful metric to assert: with your feedback, we'll find out over time if it is or is not.actual: 3 at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.assertTrue(Assert.java:42) at org.mozilla.fenix.perf.StartupExcessiveResourceUseTest.verifyRunBlockingAndStrictModeSuppresionCount(StartupExcessiveResourceUseTest.kt:92)

[AndiAJ]

Closing, as it didn't fail for approx. a month.

[Mugurell]

Saw this failing again in https://github.com/mozilla-mobile/fenix/pull/26336 - https://console.firebase.google.com/u/1/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/5801008004089086306

[AndiAJ]

💥 Failed 1x on:

► 8/17 Main

• verifyRunBlockingAndStrictModeSuppresionCount

Firebase link

java.lang.AssertionError: ConstraintLayout being a common direct descendant of a RecyclerView count does not match expected count.

This heuristic-based performance test is expected measure the number of occurrences of known
performance anti-patterns and fail when that count changes. Please read the class documentation
for more details about this test and an explanation of what the failed heuristic is expected to
measure. Please consult the performance team if you have questions.

actual: 6 at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.assertTrue(Assert.java:42) at org.mozilla.fenix.perf.StartupExcessiveResourceUseTest.verifyRunBlockingAndStrictModeSuppresionCount(StartupExcessiveResourceUseTest.kt:126)

Firebase link

java.lang.AssertionError: ConstraintLayout being a common direct descendant of a RecyclerView count does not match expected count.

This heuristic-based performance test is expected measure the number of occurrences of known
performance anti-patterns and fail when that count changes. Please read the class documentation
for more details about this test and an explanation of what the failed heuristic is expected to
measure. Please consult the performance team if you have questions.

actual: 6 at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.assertTrue(Assert.java:42) at org.mozilla.fenix.perf.StartupExcessiveResourceUseTest.verifyRunBlockingAndStrictModeSuppresionCount(StartupExcessiveResourceUseTest.kt:126)

[AndiAJ]

💥 Failed 2x on:

► 8/18 Main

• verifyRunBlockingAndStrictModeSuppresionCount

Firebase link

java.lang.AssertionError: ConstraintLayout being a common direct descendant of a RecyclerView count does not match expected count.

This heuristic-based performance test is expected measure the number of occurrences of known
performance anti-patterns and fail when that count changes. Please read the class documentation
for more details about this test and an explanation of what the failed heuristic is expected to
measure. Please consult the performance team if you have questions.

actual: 6 at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.assertTrue(Assert.java:42) at org.mozilla.fenix.perf.StartupExcessiveResourceUseTest.verifyRunBlockingAndStrictModeSuppresionCount(StartupExcessiveResourceUseTest.kt:126)

Firebase link

java.lang.AssertionError: ConstraintLayout being a common direct descendant of a RecyclerView count does not match expected count.

This heuristic-based performance test is expected measure the number of occurrences of known
performance anti-patterns and fail when that count changes. Please read the class documentation
for more details about this test and an explanation of what the failed heuristic is expected to
measure. Please consult the performance team if you have questions.

actual: 1 at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.assertTrue(Assert.java:42) at org.mozilla.fenix.perf.StartupExcessiveResourceUseTest.verifyRunBlockingAndStrictModeSuppresionCount(StartupExcessiveResourceUseTest.kt:126)

Firebase link

java.lang.AssertionError: ConstraintLayout being a common direct descendant of a RecyclerView count does not match expected count.

This heuristic-based performance test is expected measure the number of occurrences of known
performance anti-patterns and fail when that count changes. Please read the class documentation
for more details about this test and an explanation of what the failed heuristic is expected to
measure. Please consult the performance team if you have questions.

actual: 6 at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.assertTrue(Assert.java:42) at org.mozilla.fenix.perf.StartupExcessiveResourceUseTest.verifyRunBlockingAndStrictModeSuppresionCount(StartupExcessiveResourceUseTest.kt:126)

Firebase link

java.lang.AssertionError: ConstraintLayout being a common direct descendant of a RecyclerView count does not match expected count.

This heuristic-based performance test is expected measure the number of occurrences of known
performance anti-patterns and fail when that count changes. Please read the class documentation
for more details about this test and an explanation of what the failed heuristic is expected to
measure. Please consult the performance team if you have questions.

actual: 1 at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.assertTrue(Assert.java:42) at org.mozilla.fenix.perf.StartupExcessiveResourceUseTest.verifyRunBlockingAndStrictModeSuppresionCount(StartupExcessiveResourceUseTest.kt:126)

[Mugurell]

The parts that failed were disabled in https://github.com/mozilla-mobile/fenix/pull/26512. The recent failures seem like a regression from https://github.com/mozilla-mobile/fenix/pull/26336 but with no direct relation.

[cpeterso]

Moved to bugzilla: https://bugzilla.mozilla.org/show_bug.cgi?id=1807303

Change performed by the Move to Bugzilla add-on.