Waiting for app ReplicaSet be marked available indefinitely

[jsravn]

What happened?

A completely healthy deployment gets stuck on "Waiting for app ReplicaSet be marked available" despite all replicas being fully available.

Steps to reproduce

It's not clear yet to me how to reproduce reliably. It seems to happen sporadically. Also I see there are quite a few issues already that have the same problem - I guess it's not completely fixed?

Expected Behavior

It should notice the deployment is healthy and proceed.

Actual Behavior

Do you want to perform this update? yes
Updating (infra.pre-dev):
     Type                                    Name                 Status       Info
     pulumi:pulumi:Stack                     infra-infra.pre-dev  running      read kubernetes:core/v1:Secret gke-cluster-addons-feed-original-tls
     └─ core_gcp:infra:cluster_addons        gke-cluster-addons                
        ├─ core:gcp:lib:traefik              traefik-internal                  
 ~      │  └─ kubernetes:apps/v1:Deployment  traefik-internal     updating...  [diff: ~spec]; [1/2] Waiting for app ReplicaSet be marked available (3/3 Pods available)
        └─ core:gcp:lib:traefik              traefik-external                  
 ~         └─ kubernetes:apps/v1:Deployment  traefik-external     updating     [diff: ~spec]; [1/2] Waiting for app ReplicaSet be marked available (1/1 Pods available)

ReplicaSet status:

status:
  availableReplicas: 3
  fullyLabeledReplicas: 3
  observedGeneration: 11
  readyReplicas: 3
  replicas: 3

Deployment status:

status:
  availableReplicas: 3
  conditions:
  - lastTransitionTime: "2022-04-26T10:41:05Z"
    lastUpdateTime: "2022-04-26T10:41:05Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2022-04-13T09:10:26Z"
    lastUpdateTime: "2022-04-26T10:41:26Z"
    message: ReplicaSet "traefik-internal-b28bb768-6547995858" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 24
  readyReplicas: 3
  replicas: 3
  updatedReplicas: 3

Versions used

CLI          
Version      3.30.0
Go Version   go1.17.9
Go Compiler  gc

Plugins
NAME        VERSION
gcp         6.20.0
gcp         6.20.0
kubernetes  3.18.2
kubernetes  3.18.2
nodejs      unknown
random      4.4.2
random      4.4.2

Host     
OS       nixos
Version  21.11 (Porcupine)
Arch     x86_64

This project is written in nodejs (/nix/store/46g0dmf6rcpikbzs22y7w4amyg0ciksi-nodejs-16.14.2/bin/node v16.14.2)

Additional context

Tried running a pulumi refresh beforehand, but it made no difference. Kubernetes is version 1.22.

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[jsravn]

Unsure why, but after updating my pulumi-kubernetes provider to 3.18.3 (from 3.18.2) it proceeded to work.

[jsravn]

This is still happening to us. Not sure what triggers it yet.

[jsravn]

This keeps happening to us and we haven't quite figured out what triggers it. I think it may be related to performing a refresh before doing pulumi up, since it seems to happen pretty reliably when I do pulumi --refresh --skip-preview --yes up. Whereas it never seems to happen if I avoid refresh.

[Shrooblord]

Hey there! Thanks for the insight into pulumi refresh. I routinely do pulumi refresh before a pulumi up and seem to be running into the same problem. However, if I do pulumi refresh --> pulumi up --> error --> pulumi up, I still get the same errors you describe in the first post.

Is there anything you do to 'reset' pulumi so the refresh bug doesn't come up?

[thanhtoan1196]

any update?