longhorn-manager icon indicating copy to clipboard operation
longhorn-manager copied to clipboard

fix(upgrade): add sleep to allow event to flush before panic

Open james-munson opened this issue 1 year ago • 1 comments
trafficstars

Which issue(s) this PR fixes:

Issue https://github.com/longhorn/longhorn/issues/9569

What this PR does / why we need it:

Add a sleep after Run() before panicking to let the queued event get flushed. We thought that eventBroadcaster.Shutdown() would take care of that, and it did in some testing, but there is still a race and it does not work reliably. QA found that while testing the feature and I did reproduce it - see the note in https://github.com/longhorn/longhorn/issues/9569#issuecomment-2425705470

Special notes for your reviewer:

This is the simplest fix. We could do something more complicated, like calling Watch() on the events and waiting for it to appear, but there is no need to speed up the exit as soon as possible. The upgrade will not go forward anyway. The important thing is to guarantee the event can be found so the problem can be corrected.

Additional documentation or context

james-munson avatar Oct 24 '24 03:10 james-munson