Top flaky tests on `master` by frequency

[dsabsay]

The following table lists the tests that have failed on the master branch over the last couple months. These tests can be assumed to be flaky.

This data was generated by a periodic script that parses Action run logs and extracts the tests that failed. Links to the logs in GitHub Actions is included but log retention is short (I think 1 month) so will not be available for all instances.

The data below includes action runs since 2025-02-17. Last updated 2025-06-19.

---

Fixed tests:

• TestNativeHistogramFuzz (thanks @SungJin1212)
• TestSchedulerShutdown_FrontendLoop (thanks @SungJin1212)
• TestRecoverAlertsPostOutage

Name	Number of Failures	Links
TestPrometheusCompatibilityQueryFuzz	16	0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
TestQuerierWithBlocksStorageRunningInSingleBinaryMode	14	0 1 2 3 4 5 6 7 8 9 10 11 12 13
TestNativeHistogramFuzz	12	0 1 2 3 4 5 6 7 8 9 10 11
TestRing_ShuffleShardWithLookback_CorrectnessWithFuzzy	12	0 1 2 3 4 5 6 7 8 9 10 11
TestExperimentalPromQLFuncsWithPrometheus	10	0 1 2 3 4 5 6 7 8 9
TestCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning	9	0 1 2 3 4 5 6 7 8
TestParquetFuzz	6	0 1 2 3 4 5
TestMinimizeSpreadTokenGenerator	6	0 1 2 3 4 5
TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod	4	0 1 2 3
TestBackwardCompatibilityQueryFuzz	3	0 1 2
TestStoreGatewayLazyExpandedPostingsSeriesFuzz	3	0 1 2
TestRecoverAlertsPostOutage	3	0 1 2
TestQuerierWithBlocksStorageRunningInMicroservicesMode	3	0 1 2
TestExpandedPostingsCacheFuzz	3	0 1 2
TestQueryFrontendResponseSizeLimit	3	0 1 2
TestDisableChunkTrimmingFuzz	2	0 1
TestLoader_ShouldCacheIndexNotFoundOnBackgroundUpdates	2	0 1
TestPartitionCompactor_ShouldFailCompactionOnTimeout	2	0 1
TestPushRace	2	0 1
TestCompactor_RingLifecyclerShouldAutoForgetUnhealthyInstances	2	0 1

[SungJin1212]

@yeya24 It needs to be reopened since many of the flaky tests remain.

[CharlieTLe]

Logs from TestRing_ShuffleShardWithLookback_CorrectnessWithFuzzy have expired. Do we still see it occurring?

[dsabsay]

@CharlieTLe The last time that one failed was 3 months ago it looks like. I just updated the table with the latest data.