Bug: Multi Config setup Not waiting to stabilise all deployments when in different NS

[JJRdec]

trafficstars

Expected behavior

Skaffold should wait for all deployments to stabilise.

Actual behavior

Skaffold does not wait for all deployments when in different namespaces

When Cloud Tools are used with Skaffold to do IDE debugging, its not possible on deployments that have not be waited on.

Information

• Skaffold version: v1.39.1
• Operating system: Ubuntu-20.04 (WSL)
• Installed via: skaffold.dev
• Contents of skaffold.yaml:

1. https://github.com/GoogleContainerTools/skaffold/blob/main/examples/multi-config-microservices/skaffold.yaml
2. https://github.com/GoogleContainerTools/skaffold/blob/main/examples/multi-config-microservices/leeroy-app/skaffold.yaml
3. https://github.com/GoogleContainerTools/skaffold/blob/main/examples/multi-config-microservices/leeroy-web/skaffold.yaml

Steps to reproduce the behavior

1. git clone https://github.com/GoogleContainerTools/skaffold.git
2. cd skaffold/examples/multi-config-microservices
3. kubectl create ns a
4. kubectl create ns b
5. Update skaffold/examples/multi-config-microservices/leeroy-web/kubernetes/deployment.yaml to deploy to namespace a
6. Update skaffold/examples/multi-config-microservices/leeroy-app/kubernetes/deployment.yaml to deploy to namespace b
7. skaffold dev or skaffold dev -vdebug

Log snippet

 - leeroy-app
 - leeroy-web
Generating tags...
 - base -> base:9a5b49c-dirty
 - leeroy-app -> leeroy-app:9a5b49c-dirty
 - leeroy-web -> leeroy-web:9a5b49c-dirty
Checking cache...
 - base: Found Locally
 - leeroy-app: Found Locally
 - leeroy-web: Found Locally
Tags used in deployment:
 - base -> base:e0574e1cf7d299be9629c6c53b597930df0b1de0298f554ba0a1fd0fba7b9d04
 - leeroy-app -> leeroy-app:5d0dfba8df72565108dfd6ea0a788059b840b509a43e76ab9a130b0bd74b03e7
 - leeroy-web -> leeroy-web:3898e37e4b493c4a6e1b1162420e6bf0a7cfe23381136e6654eecd7c80269b42
Starting deploy...
 - service/leeroy-app created
 - deployment.apps/leeroy-app created
 - deployment.apps/leeroy-web created
Waiting for deployments to stabilize...
 - b:deployment/leeroy-app is ready.
Deployments stabilized in 4.081 seconds

Log snippet (with -vebdug)

status: {}  subtask=-1 task=DevLoop
DEBU[0006] Running command: [kubectl --context docker-desktop get -f - --ignore-not-found -ojson]  subtask=1 task=Deploy
DEBU[0006] Command output: []                            subtask=1 task=Deploy
DEBU[0006] 1 manifests to deploy. 1 are updated or new   subtask=1 task=Deploy
DEBU[0006] Running command: [kubectl --context docker-desktop apply -f -]  subtask=1 task=Deploy
 - deployment.apps/leeroy-web created
INFO[0006] Deploy completed in 1.119 second              subtask=-1 task=Deploy
Waiting for deployments to stabilize...
DEBU[0006] getting client config for kubeContext: `docker-desktop`  subtask=-1 task=DevLoop
DEBU[0006] getting client config for kubeContext: `docker-desktop`  subtask=-1 task=DevLoop
DEBU[0006] checking status b:deployment/leeroy-app       subtask=-1 task=Deploy
DEBU[0007] Running command: [kubectl --context docker-desktop rollout status deployment leeroy-app --namespace b --watch=false]  subtask=-1 task=Deploy
DEBU[0007] Command output: [Waiting for deployment "leeroy-app" rollout to finish: 0 of 1 updated replicas are available...
]  subtask=-1 task=Deploy
DEBU[0007] Pod "leeroy-app-54f6db497-4p95p" scheduled but not ready: checking container statuses  subtask=-1 task=DevLoop
DEBU[0007] Fetching events for pod "leeroy-app-54f6db497-4p95p"  subtask=-1 task=DevLoop
DEBU[0008] Running command: [kubectl --context docker-desktop rollout status deployment leeroy-app --namespace b --watch=false]  subtask=-1 task=Deploy
DEBU[0008] Command output: [Waiting for deployment "leeroy-app" rollout to finish: 0 of 1 updated replicas are available...
]  subtask=-1 task=Deploy
DEBU[0008] Pod "leeroy-app-54f6db497-4p95p" scheduled but not ready: checking container statuses  subtask=-1 task=DevLoop
DEBU[0008] Fetching events for pod "leeroy-app-54f6db497-4p95p"  subtask=-1 task=DevLoop
DEBU[0009] Running command: [kubectl --context docker-desktop rollout status deployment leeroy-app --namespace b --watch=false]  subtask=-1 task=Deploy
DEBU[0009] Command output: [deployment "leeroy-app" successfully rolled out
]  subtask=-1 task=Deploy
 - b:deployment/leeroy-app is ready.
Deployments stabilized in 3.081 seconds

[JJRdec]

Just to add, when Skaffold is used in conjunction with Cloud Code in say Intellij, logs are not streamed and debugging is not possible on deployments that aren't waited on.

[tejal29]

Sorry @JJRdec for this experience.

We will have someone from the team look into it. It could be that namespace a was not registered. This looks like a very rare use case and hence lower priority issue for us.

Please feel free to comment and let me know if this is a common practice.

[ericzzzzzzz]

Tested in v2, logs from deployments in different ns are streamed. I'm going to close it, feel free to re-open if the issue still exists.