helm_remote: Wait for namespace to exist before moving on

[tomislater]

Code:

helm_remote('kafka',
  repo_name='bitnami',
  repo_url='https://charts.bitnami.com/bitnami',
  version='12.17.3',
  release_name='kafka',
  namespace='kafka',
  create_namespace=True,
  set=[
    'deleteTopicEnable=true',
    'defaultReplicationFactor=1',
    'offsetsTopicReplicationFactor=1',
  ]
)
k8s_resource(
  'kafka',
  labels=["databases"]
)
k8s_resource(
  'kafka-zookeeper',
  labels=["databases"]
)

Error:

STEP 1/1 — Deploying
     Applying YAML to cluster

Build Failed: namespaces "kafka" not found

It seems that sometimes it works and sometimes does not. When I triggered update via UI (on kafka resource) it worked. But , the first time when I saved Tiltfile the error occurred.

[lianmakesthings]

Hi @tomislater I've checked the helm_remote extension, and it should create the namespace before building anything. (https://github.com/tilt-dev/tilt-extensions/blob/master/helm_remote/Tiltfile#L139). My best guess atm is that your cluster hasn't finished creating the namespace by the time the build starts, which would explain why it works on the second try, when you run helm_remote from the UI.

Unfortunately, I wasn't able to reproduce the issue on my own machine. What does your tilt doctor output say?

[tomislater]

Hi @lianmakesthings

Yeah, I also have checked the code and it looks that sometimes k8s needs more time to create a namespace thus the error. I should use something like wait_for and wait for creating a namespace.

[lianmakesthings]

Yeah, you could use the namespace extension to create the namespace yourself and then wait. https://github.com/tilt-dev/tilt-extensions/tree/master/namespace

Although, I do believe it would be worth implementing waiting for namespace creation in the helm_remote extension itself.

[nicks]

Is there anything missing from your repro case here? I tried it out with tilt ci and got this output:

uncategorized │ 
uncategorized │ Initial Build • uncategorized
uncategorized │ STEP 1/1 — Deploying
uncategorized │      Applying YAML to cluster
uncategorized │      Objects applied to cluster:
uncategorized │        → kafka:namespace
uncategorized │        → kafka:serviceaccount
uncategorized │        → kafka-scripts:configmap
uncategorized │ 
uncategorized │      Step 1 - 0.30s (Deploying)
uncategorized │      DONE IN: 0.30s

i.e., that there's usually a list of resources it's applying in the output (and there's not in your repro case, which might be a pointer to something going wrong).

usually tilt is pretty smart about applying the namespace before other resources. There also might be something wrong with Tilt's dependency ordering. But we also haven't heard any reports like this recently (despite a ton of users who use tilt to install Kafka), so I'm wondering if there's something unique about your dev env that might be triggering this bug.

[tomislater]

@nicks I try to debug this case today and provide more info.

[tomislater]

Btw, kafka was only an example. I can reproduce this with other charts also. I am still debugging this.

[tomislater]

My Tiltfile:

version_settings(constraint='>=0.23.5')

load('ext://namespace', 'namespace_create', 'namespace_inject')
load('ext://local_output', 'local_output')
load('Tiltfile-dockerfile', 'dockerfile_contents')

secret_settings(disable_scrub=True)

include('Tiltfile-databases')
...

And in this Tiltfile-databases I put this code:

helm_remote('redis',
  repo_name='bitnami',
  repo_url='https://charts.bitnami.com/bitnami',
  version='15.4.0',
  release_name='test-redis',
  namespace='test-redis',
  create_namespace=True,
  set=[
    'architecture=standalone',
    'auth.enabled=false',
    'master.readinessProbe.enabled=false',
    'master.livenessProbe.enabled=false',
    'master.persistence.enabled=false',
  ]
)

I use tilt up --debug --verbose --klog 0 and in dashboard for this resource I see this:

Building with BuildOrder: BuildOrder{ *buildcontrol.LiveUpdateBuildAndDeployer *buildcontrol.DockerComposeBuildAndDeployer *buildcontrol.ImageBuildAndDeployer *buildcontrol.LocalTargetBuildAndDeployer }
Trying to build and deploy with *buildcontrol.LiveUpdateBuildAndDeployer

Falling back to next update method…
REASON: no targets for Live Update found

Trying to build and deploy with *buildcontrol.DockerComposeBuildAndDeployer

Falling back to next update method…
REASON: DockerComposeBuildAndDeployer does not support target type model.K8sTarget

Trying to build and deploy with *buildcontrol.ImageBuildAndDeployer
STEP 1/1 — Deploying
     No images injected into Kubernetes YAML
     No images injected into Kubernetes YAML
     No images injected into Kubernetes YAML
     Applying YAML to cluster

Build Failed: namespaces "test-redis" not found

Additionally, because my laptop does not have enough cpu/ram to run docker and 4k monitor... I set up k8s cluster via k3d on my local PC and I use local network to reach my PC from my laptop. Tilt use the right context of course.

.kube/config:

...
- cluster:
    certificate-authority-data: ***hidden***
    server: https://192.168.1.6:35027
  name: k3d-local-cluster
...

In uncategorized tab, I found this:

Trying to build and deploy with *buildcontrol.ImageBuildAndDeployer
STEP 1/1 — Deploying
     No images injected into Kubernetes YAML
     No images injected into Kubernetes YAML
     ........ xx times the same message ........
     No images injected into Kubernetes YAML
     Applying YAML to cluster
     Objects applied to cluster:
              → test-redis:namespace
              → test-redis:serviceaccount
              → test-redis-configuration:configmap
              → test-redis-health:configmap
              → test-redis-scripts:configmap

The namespace is created after this action btw.

[tomislater]

With this code I can achieve the same error:

namespace_create("test-redis")
helm_remote('redis',
  repo_name='bitnami',
  repo_url='https://charts.bitnami.com/bitnami',
  version='15.4.0',
  release_name='test-redis',
  namespace='test-redis',
  # create_namespace=True,
  set=[
    'architecture=standalone',
    'auth.enabled=false',
    'master.readinessProbe.enabled=false',
    'master.livenessProbe.enabled=false',
    'master.persistence.enabled=false',
  ]
)

Looks like a problem with ordering in Tilt? 🤔

[tomislater]

Btw, output from tilt doctor:

➜ tilt doctor
Tilt: v0.23.5, built 2022-01-07
System: darwin-amd64
---
Docker
- Host: Error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
- Compose Version: v2.2.1
---
Kubernetes
- Env: k3d
- Context: k3d-local-cluster
- Cluster Name: k3d-local-cluster
- Namespace: default
- Container Runtime: containerd
- Version: v1.21.7+k3s1
- Cluster Local Registry: {Host:localhost:5432 hostFromCluster:local-registry:5000 SingleName:}
---
Thanks for seeing the Tilt Doctor!
Please send the info above when filing bug reports. 💗

[tomislater]

Even if I refresh my Tiltfile (via UI) nothing happens. I need to trigger update (via UI also):

[tomislater]

When I use tilt ci everything works smoothly. When I use tilt up at the first time also everything works correctly. (I manually removed test-redis namespace).

But when I comment the whole section (tilt up is running), tilt refreshes itself and removes test-redis namespace from cluster (works correctly). After that I uncomment the commented section, tilt refreshes itself, but now it has the problem.

So, it looks like a problem with live updates? 🤔

[tomislater]

How to reproduce it:

1. run tilt up
2. add helm_remote with create_namespace=True to your Tiltfile
3. save a file
4. wait
5. you should get an error: Build Failed: namespaces "<your_namespace>" not found

I can also get this error using namespace_create. So, it looks like a problem with Tilt.

[nicks]

ooooooh the k3d info is interesting.

K3d tries really hard to be fast. Sometimes they make a mistake and say they're done doing something when it's still ongoing in the background. But that's just a hunch...will try to reproduce with a k3d cluster.

[amontalban]

I'm interested in this as well, mainly having a way for helm_remote to wait for other resources to be ready. For example, we have a helm_remote for PostgreSQL and Redis and we would like that services that depend on these to only start after the aforementioned are ready.

I tried with wait_for_it but didn't do what I needed.

[argais]

Did this get anywhere? I am still running into this issue.

[nicks]

No one's been able to come up with a repro case. tilt already confirms with the apiserver that the namespace exists before applying other resources, so it's not clear what additional thing tilt should wait on.