argo-cd Getting "No operation is in progress" when trying to Terminate sync

Checklist:

[x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
[x] I've included steps to reproduce the bug.
[x] I've pasted the output of argocd version.

Describe the bug

The UI and the API say a sync is happening:

Screenshot 2024-02-08 at 3 23 30 PM Screenshot 2024-02-08 at 3 27 56 PM

But when I try terminate, it gives me this error

Unable to terminate operation: Unable to terminate operation. No operation is in progress

Screenshot 2024-02-08 at 3 24 12 PM

This is happening across all our clusters. I see nothing in the logs

To Reproduce

All I do is try to Terminate a running sync. Same error through CI and UI.

I tried re-creating the app. I've tried deleting application controller pods. It doesn't help.

Expected behavior

I expect the sync get terminated

Screenshots

See above

Version

➜  argocd version
argocd: v2.9.3+6eba5be.dirty
  BuildDate: 2023-12-02T00:45:51Z
  GitCommit: 6eba5be864b7e031871ed7698f5233336dfe75c7
  GitTreeState: dirty
  GoVersion: go1.21.4
  Compiler: gc
  Platform: darwin/amd64
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
argocd-server: v2.9.5+f943664
  BuildDate: 2024-01-19T17:57:57Z
  GitCommit: f9436641a616d277ab1f98694e5ce4c986d4ea05
  GitTreeState: clean
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
  Helm Version: v3.13.2+g2a2fb3b
  Kubectl Version: v0.24.17
  Jsonnet Version: v0.20.0

Logs

➜  argocd app get vkms
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
Name:               argocd/vkms
Project:            streaming
Server:             https://kubernetes.default.svc
Namespace:          default
URL:                https://<argo-domain>/applications/vkms
Repo:               <argo-repo>
Target:             main
Path:              <argo-path>
SyncWindow:         Sync Allowed
Sync Policy:        Automated
Sync Status:        OutOfSync from main (51e98a1)
Health Status:      Healthy

CONDITION  MESSAGE                                                                                                                                 LAST TRANSITION
SyncError  Failed sync attempt to 51e98a1cd67fed6e1b954cfa3fbecf95c8909913: waiting for healthy state of autoscaling/HorizontalPodAutoscaler/vkms  2024-02-08 15:11:47 -0800 PST


GROUP        KIND                     NAMESPACE  NAME                 STATUS     HEALTH   HOOK  MESSAGE
             ConfigMap                default    vkms-config-3499602  OutOfSync                 ignored (requires pruning)
             ServiceAccount           default    vkms                 OutOfSync                 serviceaccount/vkms unchanged
             ConfigMap                default    vkms-config-de552ac  OutOfSync                 configmap/vkms-config-de552ac unchanged
apps         Deployment               default    vkms                 OutOfSync  Healthy        deployment.apps/vkms configured
argoproj.io  Rollout                  default    vkms                 OutOfSync  Healthy        rollout.argoproj.io/vkms unchanged
             Service                  default    vkms                 OutOfSync  Healthy        service/vkms unchanged
autoscaling  HorizontalPodAutoscaler  default    vkms                 OutOfSync  Healthy

➜  argocd app terminate-op vkms
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
FATA[0000] rpc error: code = InvalidArgument desc = Unable to terminate operation. No operation is in progress

Feb 08 '24 23:02 teddy-wahle

Can you provide the pod logs for server and application controller pods after reproducing the issue?

Feb 09 '24 05:02 jgwest

@jgwest Here are logs from the two application controller pods and the two server pods. To create these logs I deleted all server and app controller pods so that all the logs were fresh. Then I went to UI and tried to terminate a sync in an app called vconductor and got the same error:

argocd-application-controller-1.txt argocd-application-controller-0.txt

argocd-server-55464bc74-zmhvr.txt argocd-server-55464bc74-jw67s.txt

Screenshot 2024-02-12 at 10 09 04 AM

You can see this in the logs:

time="2024-02-12T18:02:33Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = Unable to terminate operation. No operation is in progress" grpc.code=InvalidArgument grpc.method=TerminateOperation grpc.service=application.ApplicationService grpc.start_time="2024-02-12T18:02:33Z" grpc.time_ms=8.452 span.kind=server system=grpc

Feb 12 '24 18:02 teddy-wahle

This might be because of sharding functionality which is known issue in 2.10.0 version.

PR is raised for this. https://github.com/argoproj/argo-cd/issues/17124

Feb 13 '24 13:02 AnubhavSabarwal

we are on 2.10.5 and still facing this issue.

May 03 '24 18:05 morawat

I'm seeing this behaviour on v2.10.9+c071af8 too

May 06 '24 08:05 NPastorale

It is still present on v2.11.0+d3f33c0

May 16 '24 21:05 NPastorale

Is there a way to upvote and get some traction on this?

May 20 '24 17:05 morawat

This has been a problem since the inception of ArgoCD. When will we get a workaround ??

Jun 12 '24 12:06 jonra82

same issue on v2.11.4+e1284e1 I often see this issue in such scenario: A deployment is already in the cluster but we are onboarding it to ArgoCD for the first time. It has the annotation "argocd.argoproj.io/sync-options: Replace=true". I guess this "replace" behavior causes some bug, leading to sync error like "waiting for healthy state of apps/Deployment/..."