Virtualised microk8s registry settings not correctly inferred.

[dspasojevic]

Expected Behavior

Tilt correctly infers both the Host and hostFromCluster values of a registry when using microk8s in a vm. The Host should probably be the address of the registry, as addressable from the host machine (i.e. the IP address of the VM) and hostFromCluster should be localhost.

Current Behavior

The Host value is hard-coded to be localhost: https://github.com/tilt-dev/tilt/blob/77ae37f8f66c4b2d6df5eb526bd71954be73ecf2/internal/k8s/registry.go#L101

Context

tilt doctor Output

$ tilt doctor
Tilt: v0.23.9, built 2022-01-27
System: darwin-arm64
---
Docker
- Host: [default]
- Server Version: 20.10.12
- API Version: 1.41
- Builder: 2
- Compose Version: v2.2.3
---
Kubernetes
- Env: microk8s
- Context: microk8s
- Cluster Name: microk8s-cluster
- Namespace: default
- Container Runtime: containerd
- Version: v1.22.5-3+2cf820a97417ce
- Cluster Local Registry: {Host:localhost:32000 hostFromCluster: SingleName:}
---

About Your Use Case

I use macos. Docker is running in a VM using Docker Desktop. microk8s is running in a different one using multipass. I have enabled microk8s local registry. In the Docker VM, localhost (the inferred registry host) is referring to the docker vm, not the host vm. default_registry (as discussed in #4125) can not be used to override the inferred registry values.

When I hack DecideRegistry to always use the default_registry value, things work, so I think my issue is only related to the inference of the registry settings for microk8s.

[nicksieger]

Did you run microk8s install to build your cluster, or did you do any additional configuration? I want to make sure I know how to recreate the issue.

[nicks]

fwiw, it's hard-coded to localhost in microk8s too: https://github.com/ubuntu/microk8s/blob/c70f5f25b1ecb5a7d75718ee3169e967610265c1/microk8s-resources/actions/registry.yaml#L77

if you do:

kubectl edit -n kube-public configmap local-registry-hosting

and change it to the VM IP, I think Tilt will pick it up the new value automatically the next time you start tilt? then you don't need to put cluster-specific settings in the tiltfile at all.

it probably needs to look something like:

apiVersion: v1
kind: ConfigMap
metadata:
  name: local-registry-hosting
  namespace: kube-public
data:
  localRegistryHosting.v1: |
    help: "https://microk8s.io/docs/registry-built-in"
    host: "[YOUR VM IP]:32000"
    hostFromContainerRuntime: "localhost:32000"

[dspasojevic]

@nicksieger I installed a vanilla version of microk8s and enabled the registry add-on.

[dspasojevic]

@nicks nice 🎉 , after that change, my doctor output is:

Tilt: v0.23.9, built 2022-01-27
System: darwin-arm64
---
Docker
- Host: [default]
- Server Version: 20.10.12
- API Version: 1.41
- Builder: 2
- Compose Version: v2.2.3
---
Kubernetes
- Env: microk8s
- Context: microk8s
- Cluster Name: microk8s-cluster
- Namespace: default
- Container Runtime: containerd
- Version: v1.22.5-3+2cf820a97417ce
- Cluster Local Registry: {Host:192.168.64.2:32000 hostFromCluster:localhost:32000 SingleName:}
---

and the image is tagged and pushed correctly.

[nicksieger]

Unfortunately no mention of this technique here: https://microk8s.io/docs/registry-built-in#what-if-microk8s-runs-inside-a-vm-2 It's the right fix, just wondering what the best way to surface it? Make ctlptl create cluster microk8s --registry detect the VM IP and add it? Add microk8s+docker desktop detection to the registry detection code?

[dspasojevic]

With my setup, it isn't actually hitting the microk8s detection code - it is pulling from the config map. So adding docker desktop + microk8s specific detection may be unnecessary complication. Given microk8s creates the config map when enabling the registry, I'm not sure why there is microk8s detection code at all.

Personally, I would have found the documentation if it was described here (having read that a few times) and been able to solve the problem myself.

[nicks]

ya, i agree we should update that doc with some info about local-registry-hosting and how to edit it

the background here: default_registry was originally written for a world where each user would have their own personal registry for development, but a shared cluster. And so it made sense to be able to script it per-user in the tiltfile.

Over the years, the kubernetes community has moved away from that kind of setup. The far more common setup these days is either:

1. each user has their own local cluster, or
2. each user has their own namespace in a shared cluster with a shared registry

and in both cases, it makes way more sense for the registry info to be stored on the cluster itself.

[dspasojevic]

Maybe updating the API documentation for https://docs.tilt.dev/api.html#api.default_registry to make it clear that it will not override the inferred settings would be good as well.

Emitting a run time warning message if the default_registry will be ignored because a registry was detected might also be good?

[lazzarello]

Found my way here through a WSL 2 Ubuntu distro (which now supports systemd) and microk8s. It seems like the latest releases have a similar problem but the microk8s registry has changes because that configmap referenced from version 1.22 is no longer present. So I don't know how to update the cluster to expose the registry and tilt gets timeout errors.

I switched out the backend to Kind with a registry on a static port and it worked first try.

[milas]

@lazzarello AFAICT the local-registry-hosting ConfigMap still exists in latest HEAD for microk8s: addons/registry/registry.yaml (notice that it's now in canonical/microk8s-core-addons!)

See my comment over here for some more details: https://github.com/tilt-dev/tilt/pull/6001#issuecomment-1370122382