argoproj
fix: Don't try watching items with missing resourceVersion
Watching fails for resources missing a resourceVersion.
The watch is then retried again a second later, ad infinitum.
This leads to extra resource use, and an obscene amount of noise in the logs.
When resourceVersion is missing, this is normally because it's not a real object, but rather an interface to get some information from an api extension - e.g. Antrea uses this pattern, as shown in the example log below.
This issue appears to be the same thing: argoproj/argo-cd#21092
This merge request changes the behaviour for missing resourceVersion to wait for the next resync interval before trying again.
I figure this should be fine as these resources are already not working.
If you have any concerns or would like me to go about this another way (e.g. by implementing backoff instead), let me know and I'll update the MR.
Example of log illustrating the issue
The following is a three second excerpt from a cluster with multiple resources affected by this.
time="2025-03-14T09:17:47Z" level=info msg="Start watch EgressGroup.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T09:17:47Z" level=info msg="Failed to watch EgressGroup.controlplane.antrea.io on https://k8s.example.com: initial RV \"\" is not supported due to issues with underlying WATCH, retrying in 1s" server="https://k8s.example.com"
time="2025-03-14T09:17:47Z" level=info msg="Start watch SupportBundleCollection.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T09:17:47Z" level=info msg="Failed to watch SupportBundleCollection.controlplane.antrea.io on https://k8s.example.com: initial RV \"\" is not supported due to issues with underlying WATCH, retrying in 1s" server="https://k8s.example.com"
time="2025-03-14T09:17:47Z" level=info msg="Start watch AppliedToGroup.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T09:17:47Z" level=info msg="Failed to watch AppliedToGroup.controlplane.antrea.io on https://k8s.example.com: initial RV \"\" is not supported due to issues with underlying WATCH, retrying in 1s" server="https://k8s.example.com"
time="2025-03-14T09:17:47Z" level=info msg="Start watch NetworkPolicy.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T09:17:47Z" level=info msg="Failed to watch NetworkPolicy.controlplane.antrea.io on https://k8s.example.com: initial RV \"\" is not supported due to issues with underlying WATCH, retrying in 1s" server="https://k8s.example.com"
time="2025-03-14T09:17:47Z" level=info msg="Start watch AddressGroup.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T09:17:47Z" level=info msg="Failed to watch AddressGroup.controlplane.antrea.io on https://k8s.example.com: initial RV \"\" is not supported due to issues with underlying WATCH, retrying in 1s" server="https://k8s.example.com"
time="2025-03-14T09:17:48Z" level=info msg="Start watch EgressGroup.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T09:17:48Z" level=info msg="Failed to watch EgressGroup.controlplane.antrea.io on https://k8s.example.com: initial RV \"\" is not supported due to issues with underlying WATCH, retrying in 1s" server="https://k8s.example.com"
time="2025-03-14T09:17:48Z" level=info msg="Start watch SupportBundleCollection.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T09:17:48Z" level=info msg="Failed to watch SupportBundleCollection.controlplane.antrea.io on https://k8s.example.com: initial RV \"\" is not supported due to issues with underlying WATCH, retrying in 1s" server="https://k8s.example.com"
time="2025-03-14T09:17:48Z" level=info msg="Start watch AppliedToGroup.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T09:17:48Z" level=info msg="Failed to watch AppliedToGroup.controlplane.antrea.io on https://k8s.example.com: initial RV \"\" is not supported due to issues with underlying WATCH, retrying in 1s" server="https://k8s.example.com"
time="2025-03-14T09:17:48Z" level=info msg="Start watch NetworkPolicy.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T09:17:48Z" level=info msg="Failed to watch NetworkPolicy.controlplane.antrea.io on https://k8s.example.com: initial RV \"\" is not supported due to issues with underlying WATCH, retrying in 1s" server="https://k8s.example.com"
time="2025-03-14T09:17:48Z" level=info msg="Start watch AddressGroup.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T09:17:48Z" level=info msg="Failed to watch AddressGroup.controlplane.antrea.io on https://k8s.example.com: initial RV \"\" is not supported due to issues with underlying WATCH, retrying in 1s" server="https://k8s.example.com"
time="2025-03-14T09:17:49Z" level=debug msg="Ignoring change of object because none of the watched resource fields have changed" api-version=external-secrets.io/v1beta1 kind=ExternalSecret name=dev-starfleet-portal-envs namespace=team-starfleet server="https://k8s.example.com"
time="2025-03-14T09:17:49Z" level=info msg="Start watch EgressGroup.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T09:17:49Z" level=info msg="Failed to watch EgressGroup.controlplane.antrea.io on https://k8s.example.com: initial RV \"\" is not supported due to issues with underlying WATCH, retrying in 1s" server="https://k8s.example.com"
time="2025-03-14T09:17:49Z" level=info msg="Start watch SupportBundleCollection.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T09:17:49Z" level=info msg="Failed to watch SupportBundleCollection.controlplane.antrea.io on https://k8s.example.com: initial RV \"\" is not supported due to issues with underlying WATCH, retrying in 1s" server="https://k8s.example.com"
time="2025-03-14T09:17:49Z" level=info msg="Start watch AppliedToGroup.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T09:17:49Z" level=info msg="Failed to watch AppliedToGroup.controlplane.antrea.io on https://k8s.example.com: initial RV \"\" is not supported due to issues with underlying WATCH, retrying in 1s" server="https://k8s.example.com"
time="2025-03-14T09:17:49Z" level=info msg="Start watch NetworkPolicy.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T09:17:49Z" level=info msg="Failed to watch NetworkPolicy.controlplane.antrea.io on https://k8s.example.com: initial RV \"\" is not supported due to issues with underlying WATCH, retrying in 1s" server="https://k8s.example.com"
time="2025-03-14T09:17:49Z" level=info msg="Start watch AddressGroup.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T09:17:49Z" level=info msg="Failed to watch AddressGroup.controlplane.antrea.io on https://k8s.example.com: initial RV \"\" is not supported due to issues with underlying WATCH, retrying in 1s" server="https://k8s.example.com"
Example of log with this MR applied
Note: Only contains lines containing Deployment and NetworkPolicy.controlplane.
time="2025-03-14T12:16:56Z" level=info msg="Start watch NetworkPolicy.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T12:16:57Z" level=info msg="Start watch Deployment.apps on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T12:16:58Z" level=info msg="Ignoring watch for NetworkPolicy.controlplane.antrea.io on https://k8s.example.com due to missing resourceVersion" server="https://k8s.example.com"
time="2025-03-14T12:26:57Z" level=info msg="Failed to watch Deployment.apps on https://k8s.example.com: Resyncing Deployment.apps on https://k8s.example.com due to timeout, retrying in 1s" server="https://k8s.example.com"
time="2025-03-14T12:26:57Z" level=info msg="Start watch Deployment.apps on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T12:26:58Z" level=info msg="Failed to watch NetworkPolicy.controlplane.antrea.io on https://k8s.example.com: Resyncing NetworkPolicy.controlplane.antrea.io on https://k8s.example.com due to timeout, retrying in 1s" server="https://k8s.example.com"
time="2025-03-14T12:26:58Z" level=info msg="Start watch NetworkPolicy.controlplane.antrea.io on https://k8s.example.com" server="https://k8s.example.com"
time="2025-03-14T12:26:58Z" level=info msg="Ignoring watch for NetworkPolicy.controlplane.antrea.io on https://k8s.example.com due to missing resourceVersion" server="https://k8s.example.com"
Codecov Report
Attention: Patch coverage is 0% with 15 lines in your changes missing coverage. Please review.
Project coverage is 53.33%. Comparing base (
8849c3f) to head (8d347a2). Report is 35 commits behind head on master.
| Files with missing lines | Patch % | Lines |
|---|---|---|
| pkg/cache/cluster.go | 0.00% | 14 Missing and 1 partial :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## master #704 +/- ##
==========================================
- Coverage 54.26% 53.33% -0.94%
==========================================
Files 64 64
Lines 6164 6433 +269
==========================================
+ Hits 3345 3431 +86
- Misses 2549 2726 +177
- Partials 270 276 +6
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
🚀 New features to boost your workflow:
- ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
Quality Gate passed
Issues
0 New issues
0 Accepted issues
Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code
See analysis details on SonarQube Cloud
![sonarqubecloud[bot] avatar](https://avatars.githubusercontent.com/in/12526?v=4 "Published by a pub.dev verified publisher"){kind=small}
When resourceVersion is missing, this is normally because it's not a real object, but rather an interface to get some information from an api extension
Does this mean an aggregated API, or something different?
When resourceVersion is missing, this is normally because it's not a real object, but rather an interface to get some information from an api extension
Does this mean an aggregated API, or something different?
Yeah that's it, sorry for the imprecise language. Guess I should also qualify that when I say "normally" I mean from subjective experience. That's the only cases I've seen it.
If I read the Kubernetes API Conventions document right I think it's probably safe to assume only aggregated APIs will ever return resources without resourceVersion, but I'm not certain.
@sivchari Added tests now. I didn't find any sensible places to hook the tests into, so I used the log messages. I hope that's ok even if it's a bit nasty. As far as I could see it was either that or go refactor a bunch of unrelated code ...
EDIT: Oops, guess something broke. Looking at it ... EDIT2: There we go.
Quality Gate passed
Issues
3 New issues
0 Accepted issues
Measures
0 Security Hotspots
0.0% Coverage on New Code
6.1% Duplication on New Code
See analysis details on SonarQube Cloud
I'll check later. If you won't be able to see my reply in these days, I'd love to be pinged from you :)
The gitops-engine repository is migrating to https://github.com/argoproj/argo-cd.
Can you please update the PR by resolving the conflicts / If it's not relevant anymore, feel free to close it You can always re-open it when the migration is over
