kind
kind copied to clipboard
kubernetes-sigs
Allow use of environment variables in kind configuration
What would you like to be added:
It would be great if it was possible to reference environment variables in the configuration when setting up a kind cluster - eg - see the reference to PATH_TO_LOCAL_VOLUME below:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraMounts:
- hostPath: ${PATH_TO_LOCAL_VOLUME:-/tmp/local-volume-for-dev}
containerPath: /tmp/local-volume-for-dev
Why is this needed:
I am using kind for development and testing in my CI pipeline (TeamCity). For development the default path /tmp/local-volume-for-dev works fine but, for CI, I would like to set the hostPath to the check-out folder, so that references to previous builds are not left around.
In this particular case I would recommend a relative path which will be relative to where the binary is invoked, you should invoke it from the repo.
In the general case, interpolating environment variables is something I think we can best leave to a trivial shell script.
There are other tools that could be combined to help with this as well. Take a look at something like ytt. Maybe not as simple as setting environment variables, but can be pretty powerful if you get in to more complex scenarios.
I find it is a nice feature for CIs and automation, you can use other tools as Sean say, but it always ends adding complexity to your CI, I had to suffer this several times in different projects ... I've found this project https://github.com/drone/envsubst that can be useful, I'm +1 if the cost is small
Right now we perform a script yaml parse. I think an external tool like envsubst should do this, I can't recall any examples of this being baked in and definitely not in the Kubernetes project. e.g. kubectl does not support this.
Docker-compose was the tool I was comparing to which has support for env substitution with defaults
On Tue, 19 Apr 2022 at 16:19, Benjamin Elder @.***> wrote:
Right now we perform a script yaml parse. I think an external tool like envsubst should do this, I can't recall any examples of this being baked in and definitely not in the Kubernetes project. e.g. kubectl does not support this.
— Reply to this email directly, view it on GitHub https://github.com/kubernetes-sigs/kind/issues/2701#issuecomment-1102782518, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNNK35LXVMLKOTZY3GWLALVF3FJFANCNFSM5R6BPJUA . You are receiving this because you authored the thread.Message ID: @.***>
-- James Hargreaves Director, Qoobic Limited - Registered Company Number 06350337 Mobile: 07899 872 306 E-mail: @.***
Is this simple?
https://github.com/docker/compose/commit/0c27fd62364173c8862e7c05c26a3c76c65b6c2f
@aojea that is not environment variables from the process being used in config, that's environment variables of the containers
Docker-compose was the tool I was comparing to which has support for env substitution with defaults
ACK -- we've explicitly said before that we do not want to be docker-compose like though, kind creates clusters, it is not a workflow engine or config management.
The current config scheme is based on the kubernetes "component config" approach, at one point we actually used the Kubernetes api-machinery for this. This is how kuebelet, kubeadm, kops, and others in the project handle config, and it expects a yaml object similar to a kubernetes API type. They do not support environment variables
I think the most kind-native approach to this right now is to just simply use a relative path and run kind in a different location in CI and locally (or better yet -- both from the checkout)
Also encourage having identical config in local and CI (which can be accomplished with a relative ./. path + run kind from the source checkout, I think).
Supported approach: invoke kind create cluster with a minimal shell script and pass the variable expanded config that way.
You can also generate configuration structs from go code using the exported APIs.
With little demand we're not going to add new dependencies and bake in functionality that can be readily accomplished externally
Hello,
@BenTheElder have you got an example for a shell script with invoke?
We also want to use environment variables in the configuration. In our use case, we are developing custom solutions in the industry and we want to set up a Kubernetes cluster separately for each customer an project during development. This can be done very well with kind. We have a git repo for each customer and the kind configuration is also part of the repository. So every developer which works on this project can setup the same development cluster.
Our images are located at a private registry, so we need credentials to access it. This is described in https://kind.sigs.k8s.io/docs/user/private-registries/#mount-a-config-file-to-each-node. Now, however, the absolute path to the default docker credentials must be specified, i.e. /home/user/.docker/config.json. Where of course user is different for each user. It would be nicer if we could just use $HOME/.docker/config.json or ~/.docker/config.json, then it would work right away for every developer.
This is what I came up with:
kind-create-cluster-envsubst.sh
#!/bin/sh
# This script substitutes environment variables in kind-config.yaml and creates a cluster with the modified config.
# Kind does not support this functionality natively, so we have to use envsubst.
# See https://github.com/kubernetes-sigs/kind/issues/2701.
#
# Usage: ./kind-create-cluster-envsubst.sh
set -e
tempfile=$(mktemp)
cat kind-config.yaml | envsubst > $tempfile
kind create cluster --config $tempfile
rm $tempfile
