Frequent unit test failure - TestBFTDeliverer_CensorshipMonitorEvents

[denyeart]

Current Status

This is the most common unit test flake failure these days:

2024-05-02T13:09:22.8633466Z --- FAIL: TestBFTDeliverer_CensorshipMonitorEvents (1.97s)
2024-05-02T13:09:22.8634562Z     --- FAIL: TestBFTDeliverer_CensorshipMonitorEvents/repeated_censorship_events,_with_exponential_backoff (1.44s)
2024-05-02T13:09:22.8635831Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 1
2024-05-02T13:09:22.8636636Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 2
2024-05-02T13:09:22.8637428Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 3
2024-05-02T13:09:22.8638211Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 4
2024-05-02T13:09:22.8639031Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 5
2024-05-02T13:09:22.8639824Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 6
2024-05-02T13:09:22.8640603Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 7
2024-05-02T13:09:22.8641381Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 8
2024-05-02T13:09:22.8642159Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 9
2024-05-02T13:09:22.8643114Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 10
2024-05-02T13:09:22.8643909Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 11
2024-05-02T13:09:22.8644700Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 12
2024-05-02T13:09:22.8645516Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 13
2024-05-02T13:09:22.8646295Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 14
2024-05-02T13:09:22.8647089Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 15
2024-05-02T13:09:22.8647855Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 16
2024-05-02T13:09:22.8657603Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 17
2024-05-02T13:09:22.8658436Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 18
2024-05-02T13:09:22.8659259Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 19
2024-05-02T13:09:22.8660193Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 20
2024-05-02T13:09:22.8660952Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 21
2024-05-02T13:09:22.8661877Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 22
2024-05-02T13:09:22.8662651Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 23
2024-05-02T13:09:22.8663435Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 24
2024-05-02T13:09:22.8664209Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 25
2024-05-02T13:09:22.8665007Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 26
2024-05-02T13:09:22.8665759Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 27
2024-05-02T13:09:22.8666526Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 28
2024-05-02T13:09:22.8667293Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 29
2024-05-02T13:09:22.8668067Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 30
2024-05-02T13:09:22.8668829Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 31
2024-05-02T13:09:22.8669584Z         bft_deliverer_test.go:1108: monitor error channel returns censorship error num: 32
2024-05-02T13:09:22.8669933Z         bft_deliverer_test.go:1122: 
2024-05-02T13:09:22.8670264Z             Timed out after 1.001s.
2024-05-02T13:09:22.8670583Z             Expected
2024-05-02T13:09:22.8670857Z                 <int>: 34
2024-05-02T13:09:22.8671084Z             to equal
2024-05-02T13:09:22.8671357Z                 <int>: 33

Example: https://productionresultssa11.blob.core.windows.net/actions-results/48144904-dcf7-4d14-a281-1d12e615f17a/workflow-job-run-b8245a8d-c8fe-58cd-5361-3fdfc2c073c3/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-05-02T18%3A27%3A32Z&sig=ftHcV6n6HZbuSPrGMdt4XwbETer3%2FXGInu7IeAcSodU%3D&sp=r&spr=https&sr=b&st=2024-05-02T18%3A17%3A27Z&sv=2021-12-02

Goal

This unit test failure needs to be fixed.

Solution

This unit test failure needs to be fixed.

[denyeart]

@tock-ibm Could you take a look?

[denyeart]

@tock-ibm I had to re-open this issue since we are still seeing frequent failures with this test, e.g. https://productionresultssa0.blob.core.windows.net/actions-results/e60a1a69-df30-441c-b45c-e70884d32d7f/workflow-job-run-b8245a8d-c8fe-58cd-5361-3fdfc2c073c3/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-05-09T18%3A56%3A28Z&sig=EN%2BLghkEScQ5xV0A4Cn8yryFiiMSCztG9LDn3Gx6N2U%3D&sp=r&spr=https&sr=b&st=2024-05-09T18%3A46%3A23Z&sv=2021-12-02

[tock-ibm]

@denyeart I can't access that link

[denyeart]

@tock-ibm Ok, I'll provide the github action links... here is another occurrence from today: https://github.com/hyperledger/fabric/actions/runs/9040086759/job/24904148003?pr=4855

[denyeart]

It looks like this issue is resolved now.