Resource handler object retry logic is introducing chattiness in the event recorder because of aggregated errors

[abhat]

Following https://github.com/ovn-org/ovn-kubernetes/commit/8cba7dde86c8e1c15320d1327947f27257e0d6ab#diff-3f6ed978237c56cacb91eb947304ed9e429edf24d8d358be806093b0f061e154R956 errors that were previously suppressed (and logged at INFO level) are getting recorded as error events and this is resulting in too many events. This is expected behavior when the ovn-k node comes up, so we either need to suppress these errors as we used to https://github.com/ovn-org/ovn-kubernetes/blob/master/go-controller/pkg/retry/obj_retry.go#L670 or add logic to rate limit the event logging.

[tssurya]

cc @arghosh93 FYI

[arghosh93]

/assign @arghosh93

[arghosh93]

As it turns out errors.As() is not supported with Aggregate type error. When we are passing Aggregate type error to IsSuppressedError() function it always returns false and as a result Error gets logged in both events and container logs. Working on a PR to get around this issue and will raise it soon.