Feature request: support determining image_chroot at runtime

[abentley]

We want to be free to vary our target cluster and image registry at runtime without invalidating the build cache. So we want to use runtime-available values such as test_env or volatile-status. We reason that as long as the binary assets don't change, the choice of where to deploy them should be able to vary freely. Already, we can vary the cluster at runtime by defining our k8s_object with a specific context, and then updating that context to point to whatever cluster we desire.

k8s_object appears like a good option for varying the registry, since it supports stamp variables for image_chroot. But these values are substituted too soon-- at build time, rather than runtime. This seems unnecessarily early, since it is a runtime rule rather than a build rule.

There's a pending PR to resolve stamped values at runtime, but this relies on build-time definitions: https://github.com/bazelbuild/rules_k8s/pull/505

If we could resolve stamp variables at runtime, we would arrange to determine the target image registry based on the target cluster, and provide that as a volatile value. We'd define the k8s_object to use that as the image_chroot.

[mariusgrigoriu]

One approach you can take to handling runtime data is to implement a custom resolver, which is documented.

[abentley]

Sure, but the default resolver is already very close to what we want, so we'd probably just wrap it. And we can't see a significant advantage to the default resolver's current behaviour of evaluating these values at build time, given that they are used at runtime, and given that there's already work (#505) going into doing the resolution at runtime. So it seemed like we should start with a feature request before going our own way.