eventing
eventing copied to clipboard
knative
[flaky] test/upgrade.TestEventingUpgrades/VerifyContinualTests/EventingContinualTest
Auto-generated issue tracking flakiness of test
- Test name: test/upgrade.TestEventingUpgrades/VerifyContinualTests/EventingContinualTest
- Repository name: eventing
Latest result for this test: Flaky Last build start time: 2022-05-03 01:00:39 +0000 UTC Failed 1 times out of 5 runs. Failed runs: 1520930840517808128
2022-05-03 01:00:39 +0000 UTC: ✔ ✔ ✖ ✔ ✔ 2022-05-02 01:01:03 +0000 UTC: ✖ ✔ ✔ ✔ ✔ 2022-04-30 01:01:02 +0000 UTC: ✔ ✔ ✔ ✔ ✔ 2022-04-29 01:00:37 +0000 UTC: ✔ ✔ ✔ ✔ ✔ 2022-04-28 01:01:10 +0000 UTC: ✔ ✔ ✔ ✖ ✔ 2022-04-15 01:00:12 +0000 UTC: ✔ ✔ ✔ ✔ ✔ 2022-04-14 01:00:12 +0000 UTC: ✔ ✔ ✔ ✔ ✔ 2022-04-13 01:00:18 +0000 UTC: ✔ ✔ ✔ ✔ ✖ 2022-04-12 01:00:12 +0000 UTC: ✔ ✔ ✖ ✔ ✔ 2022-04-11 01:00:08 +0000 UTC: ✖ ✔ ✔ ✔ ✔
Older builds
2022-04-09 01:00:03 +0000 UTC: ✔ ✔ ✔ ✔ ✔ 2022-04-08 01:00:06 +0000 UTC: ✔ ✔ ✔ ✔ ✔ 2022-04-07 01:00:09 +0000 UTC: ✔ ✔ ✔ ✔ ✖ 2022-04-06 01:00:01 +0000 UTC: ✔ ✔ ✖ ✔ ✔ 2022-04-05 01:00:04 +0000 UTC: ✖ ✔ ✔ ✔ ✔ 2022-04-04 01:00:00 +0000 UTC: ✔ ✔ ✔ ✔ ✔ 2022-04-01 00:59:24 +0000 UTC: ✔ ✔ ✔ ✖ ✔ 2022-03-30 00:59:34 +0000 UTC: ✔ ✔ ✔ ✔ ✔ 2022-03-29 00:59:53 +0000 UTC: ✔ ✔ ✔ ✔ ✖ 2022-03-28 10:00:01 +0000 UTC: ✔ ✔ ✖ ✖ ✔ 2022-03-27 09:59:54 +0000 UTC: ✖ ✔ ✔ ✔ ✔ 2022-03-23 00:52:25 +0000 UTC: ✔ ✔ ✔ ✔ ✔ 2022-03-22 00:52:19 +0000 UTC: ✔ ✔ ✔ ✔ ✔ 2022-03-21 00:52:55 +0000 UTC: ✔ ✔ ✔ ✖ ✔ 2022-03-20 00:52:58 +0000 UTC: ✔ ✖ ✔ ✔ ✔ 2022-03-17 00:52:51 +0000 UTC: ✔ ✔ ✔ ✔ ✔ 2022-03-16 00:53:06 +0000 UTC: ✔ ✔ ✔ ✔ ✔ 2022-03-15 00:53:09 +0000 UTC: ✔ ✔ ✔ ✔ ✖ 2022-03-14 08:52:23 +0000 UTC: ✔ ✔ ✖ ✔ ✖ ✔ ✔ ✔ ✔ ✔ 2022-03-13 08:52:24 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 2022-03-12 08:52:24 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✖ ✖ ✔ ✖ 2022-03-11 08:52:44 +0000 UTC: ✖ ✖ ✔ ✖ ✖ ✖ ✔ ✖ ✔ ✔ 2022-03-10 08:52:51 +0000 UTC: ✔ ✖ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 2022-03-09 08:52:57 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✔ ✔ 2022-03-08 08:53:15 +0000 UTC: ✖ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 2022-03-07 08:52:52 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 2022-03-06 08:52:50 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 2022-03-05 08:53:39 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖ ✔ 2022-03-03 08:54:37 +0000 UTC: ✔ ✔ ✔ ✖ ✔ ✔ ✔ ✔ ✔ ✔ 2022-03-01 08:54:18 +0000 UTC: ✖ ✔ ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✖ 2022-02-28 08:54:01 +0000 UTC: ✔ ✖ ✔ ✖ ✖ ✔ ✔ ✔ ✔ ✔ 2022-02-25 08:53:01 +0000 UTC: ✔ ✔ ✔ ◻ ✔ ✔ ✔ ✔ ✔ ✔ 2022-02-24 08:52:50 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 2022-02-23 08:52:46 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✔ ✔ ✔ 2022-02-22 08:53:06 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 2022-02-21 08:53:12 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✖ ✔ ✔ 2022-02-20 08:53:11 +0000 UTC: ✔ ✖ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 2022-02-19 08:53:10 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 2022-02-18 08:52:44 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✔ ✔ ✖ 2022-02-17 08:52:19 +0000 UTC: ✔ ✔ ✔ ✖ ✔ ✖ ✔ ✔ ✔ ✔ 2022-02-16 08:52:53 +0000 UTC: ✔ ✔ ✔ ✔ ◻ ✔ ✖ ✔ ✔ ✔ 2022-02-15 08:53:07 +0000 UTC: ✖ ✔ ✔ ✔ ✖ ✔ ✔ ✔ ✔ ✔ 2022-02-14 08:53:02 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✔ ✔ 2022-02-13 08:53:02 +0000 UTC: ✖ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 2022-02-12 08:53:00 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 2022-02-11 08:52:58 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✔ 2022-02-10 08:52:15 +0000 UTC: ✔ ✖ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 2022-02-09 08:52:49 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✔ ✔ ✖ 2022-02-08 08:52:26 +0000 UTC: ✔ ✔ ✔ ✖ ✖ ✖ ✔ ✔ ✔ ✔ 2022-02-07 08:52:55 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✔ ✔ ✔
Looking at the failed reports we got missing events, despite we are now using retry options of:
https://github.com/knative/eventing/blob/e48c80d7f969df12aff6a7f5e7f34625f95efbbd/test/upgrade/prober/sut/broker.go#L33-L35
Stats:
- #1396207256462168064: 2 missed in 32118 events - 0.0062% chance
- #1396146857838645248: 1 missed in 39136 events - 0.0025% chance
The odds of missing events are miniscule, but they are there.
I'll try to experiment if prolonging finished wait time could help battle this flakiness. If that fails we could lower sending interval.
If either of those fail to fix flakiness, we have option to not fail on errors, as was used before.
I've looked at recent failures, and Change of this lost event is quite low, but continue to flag this test as flaky.
I think this isn't a problem of test, but Eventing fault.
/cc @vaikas @lionelvillard @aliok @devguyio
@zhongduo as the community contact this week, would you be able to take a look at this flaky test? thanks!
@cardil I remember that we believe the in memory channel will not pass the upgrade test. I am not sure how the retry can solve the problem if it is still in memory. The event missing rate is very low to me, so if we don't want to guarantee that the in-memory broker should pass the upgrade test, should we make it ignore error?
@zhongduo after recent changes (retries configured) the IMC mostly passes this test. I think we should guarantee the broker with IMC with retries should pass this test. It looks like it mostly do. Only sometimes it misses just 1-2 events, but that's in about 30k.
I think we should know what is happening to those 1-2 missed events. Where and why are we missing them?!?
@cardil I am not familiar with how retry is implemented for IMC, but I assume it is still in memory? Another thing we can probably try is to change the rollout configuration so that the deleted deployment will have some time to finish all the delivery before it is killed.
@zhongduo Yes. It is in memory. The deleted deployment should send all accepted events while doing a teardown. If it's not doing that, it's a bug.
But, I doubt that is the case here. We'd be missing a lot more events if so.