decideRegistry give auto detect priority over default registry

[RNachmanyAnyVision]

I see in the code that decideRegistry function gives priority to the local registry - https://github.com/tilt-dev/tilt/blob/07f097db95529af5460d62cebe6a1409759308ea/internal/tiltfile/tiltfile_state.go#L1013

I'm working from a remote computer using tilt and need to change the default_registry but it not possible right now. How can I "turn off" the auto-detect registry or force the default_registry()?

[nicks]

ya, if the cluster itself advertises a registry, Tilt will always give highest precedence to that.

But good news! It's just a config map in your cluster, so you can delete it or change it to tell Tilt to send the image somewhere else:

kubectl delete -n kube-public local-registry-hosting

or even better:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: local-registry-hosting
  namespace: kube-public
data:
  localRegistryHosting.v1: |
    host: "${remote_computer_ip}:${reg_port}"
    hostFromContainerRuntime: "k3d-registry.localhost:5000"
EOF

[RNachmanyAnyVision]

Thanks, @nicks. But I don't want to tell all my team to delete it every time they create a cluster.

When someone sets default_registry he should know the consequences. I think default_regitry() should be the highest precedence. In general, every manual configuration should be prioritized and override auto and default config.

Please add tilt support to the problem + I can't see any mention of this in the documentation.

[nicks]

Hmmm...default_registry is typically used by teams where each person has their own personal registry that they have write access to. It's discussed here: https://docs.tilt.dev/personal_registry.html

local-registry-hosting is typically for when the cluster itself has a preferred registry that it's optimized for. It's often used by teams that switch between remote and local clusters. It's a public spec, and is discussed here (https://docs.tilt.dev/choosing_clusters.html#kubernetes-standard-registry-discovery) and here (https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry). If you're using a personal registry for the remote cluster, you still want to use the local registry for the local cluster.

So I don't think default_registry having precedence over local-registry-hosting is the right answer here - it would break lots of projects, and would violate the spec.

I could think of other ways to fix this (e.g., having tooling in Tilt to help fix a local-registry-hosting config to be what you want).

Out of curiosity, how are your team members setting up the cluster? Are they ssh-ing into a machine and running k3d create cluster? How are you securing and exposing the api server remotely? Just trying to think through the possible steps and places where the interoperation could happen.

[RNachmanyAnyVision]

Thank @nicks

1. Each developer create k3d cluster + registry on remote machin.
2. On our laptop we set kube config to the remote cluster.
3. Then we use tilt (from the laptop) to build the images and push them to the remote cluster.

k3d v4 now has build in registry so now we need to edit the config map for the cluster instead of using tilt as we use to.

Please add tilt registry config that dont use auto detact, or add option to disable auto detect.

[nicks]

To be clear,

kubectl delete -n kube-public local-registry-hosting

is the official way to disable auto-detect. It will leave the registry running, but will tell all tools not to use the auto-detection protocol. I'm not sure it makes sense for Tilt to have a custom API for disabling the auto-detection when there's an official one.

[RNachmanyAnyVision]

I understand what you are saying but you should expect to see this problem more and more now cause of k3d 4v. It's an extra step out of tilt ecosystem, and I think you should avoid it as much as you can.

[KarlHakanNordgren]

@RNachmanyAnyVision && @nicks : I have the problem above. It looks like running

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: local-registry-hosting
  namespace: kube-public
data:
  localRegistryHosting.v1: |
    host: "${remote_computer_ip}:${reg_port}"
    hostFromContainerRuntime: "k3d-registry.localhost:5000"
EOF

fixes the problem but I agree with @RNachmanyAnyVision , why have the default_registry method if it is going to get ignored?

[realtimetodie]

@nicks why even have default_registry when it gets overriden, super annoying

[glehmann]

we could have a force=True arg in default_registry. It would default to False and allow to actually force the value even when local-registry-hosting is defined in the cluster