[Go] does not work with multiple kubeconfig files (e.g. KUBECONFIG=c1:c2)

[fabianbrash]

I removed the bash version and installed the go version on ubuntu and now I am receiving the error message "kubeconfig error: failed to load: cannot determine kubeconfig path: multiple files in KUBECONFIG are currently not supported"

[ahmetb]

Yeah this is a major point not implemented in Go version yet as I’m not fully sure how to do it in terms of modifying files when there are multiple of them (which one do you set current-context on).

I will pin this issue as others will most likely hit it too.

[fabianbrash]

so I've only been using k8s for about 4 months now so I'm still learning(probably never stop) but I'm still trying to wrap my head around the config files especially when you have more than 1, they can be merged in memory and that's what I've used I will just try and merge them all to a single file and export that to $KUBECONFIG, there is a --raw or --flatten option??

[ahmetb]

Yes, they can be merged in-memory or into a file. Read https://ahmet.im/blog/mastering-kubeconfig/. However, in this situation I need to study how to modify which file when user specifies KUBECONFIG=file1:file2:....

[fabianbrash]

thanks I'll have a look

[fabianbrash]

btw awesome work!!!

[sedooe]

Yes, they can be merged in-memory or into a file. Read https://ahmet.im/blog/mastering-kubeconfig/. However, in this situation I need to study how to modify which file when user specifies KUBECONFIG=file1:file2:....

I dug through kubectl and saw that it uses this to get the config file to write current context into.

Here I tried to add multiple kubeconfig support quickly, seems working but not sure if it's good enough for a PR or if you'd be interested with such a big one. Actually it looks bigger than it is because I had to move somethings around because of a cyclic dependency.

[ahmetb]

More context:

• Load https://github.com/kubernetes/client-go/blob/01913eaa61180521e1fd538cb9da039839008e09/tools/clientcmd/loader.go#L163

• https://github.com/kubernetes/client-go/blob/01913eaa61180521e1fd538cb9da039839008e09/tools/clientcmd/config.go#L158-L225

[hodbn]

@sedooe This is useful :) Are you planning to create a PR?

[sedooe]

@sedooe This is useful :) Are you planning to create a PR?

At that time we talked with @ahmetb and agreed on some improvements but unfortunately I couldn't find time to implement them since then. Feel free to use my fork as a baseline if you're planning to create a PR :)

[retr0h]

I use the kubecctx to manage multiple files. However, I'd rather see this in kubectx.

[bdronneau]

Hello, any updates on this behavior ?

At that time we talked with @ahmetb and agreed on some improvements but unfortunately I couldn't find time to implement them since then. Feel free to use my fork as a baseline if you're planning to create a PR :)

Do you list somewhere the improvements needed to have a PR good enough ?

Best

[willzhang]

we need:

kuectx add kubeconfig-1,kueconfig-2,kueconfig-3 > /root/.kube/config

# cat  /root/.kube/config
apiVersion: v1
kind: Config
preferences: {}
clusters:
- cluster:
    server: https://xxxxx:6443
    certificate-authority-data: xxxxx
  name: cluster1 #第一个集群名称
- cluster:
    server: https://xxxxx:6443
    certificate-authority-data: xxxxx
  name: cluster2 #第二集群名称
contexts:
- context:
    cluster: cluster1 #第一个集群名称
    user: user1 #第一个集群用户
  name: cluster1 #第一个集群名称
- context:
    cluster: cluster2 #第二集群名称
    user: user2 #第二个集群用户
  name: cluster2 #第二集群名称
users:
- name: user1 #第一个集群context中的user
  user:
    client-certificate-data: xxxxxxxxxxx
    client-key-data: xxxxxxxxxxxxxxxxxxx
- name: user2 #第二个集群context中的user
  user:
    client-certificate-data: xxxxxxxxxxx
    client-key-data: xxxxxxxxxxxxxxxxxxx

[ahmetb]

we need:

kuectx add kubeconfig-1,kueconfig-2,kueconfig-3 > /root/.kube/config

read above @willzhang:

Yes, they can be merged in-memory or into a file. Read ahmet.im/blog/mastering-kubeconfig.

[willzhang]

thanks,it's useful , but user must different in each kubeconfig

KUBECONFIG=kubeconfig1.json:kubeconfig2.json kubectl config view --merge --flatten > config
cp config /root/.kube/config

[root@master ~]# kubectx
kubernetes-admin2@disaster-cluster
kubernetes-admin@prod-cluster

[ahmetb]

thanks,it's useful , but user must different in each kubeconfig

What you want is a tool to wrangle kubeconfig. Kubectx is only for switching "context"s.

https://krew.sigs.k8s.io/plugins/ has plenty of plugins like konfig, config-registry, config-cleanup that can help you.

[tekumara]

In the meantime, could the behaviour here just be the same as the bash version? ie: just set the context on the first file.

This would at least make the go version backwards compatible, and tricks like having a temp kubeconfig file would still work.

Happy to raise a PR to do so if it seems worthwhile.

[yuzhichang]

If multiple kubeconfig are merged into one according to @willzhang , I can use kubectl config get-contexts to list contexts, kubectl config use-context to switch context.

I add following alias to ~/.bash_profile:

alias kcg="kubectl config get-contexts"
alias kcu="kubectl config use-context"

I wander whtat's the value of kubectx on contexts management?

[cprivitere]

Sure, for me, typing kctx and picking the context from a quick pop up menu is much easier than typing out the name of the context. Additionally, getting everything merged into one file doesn't work well for everyone.

[omniproc]

Yeah this is a major point not implemented in Go version yet as I’m not fully sure how to do it in terms of modifying files when there are multiple of them (which one do you set current-context on).

I will pin this issue as others will most likely hit it too.

Best effort is good enough for most cases. We assume, altought there are multiple ctx across multiple files, every one is named unique. As long as there are no conflicting names it's not an problem. So a possible strategy would be:

1. get all .kubeconfig files from ~/.kube
2. merge them into a single object kubectx works on
3. if there's no conflict switch to the selected kubectx
4. do so by coping the relevant objects to the .kube/config if they're not already there and enableing the ctx
5. if there's a conflict and the user didn't select that conflicting ctx: ignore it (just remove all conflicts from the single object generated on top)
6. if there is a conflict and the user did select the conflicting context don't do anything but fail with a error informing the user about the issue

I don't see a problem adding new, non conflicting data from other .kubeconfigs to the main config.

[icodeforyou-dot-net]

Since there appears to be not much happening here: I stumbled across this today. Very sad that the Go version doesn't implement this yet. The packaging of my distro (NixOS) no longer provides for the bash script version of kubectx. Love to see this getting added in the not so distant future.

[cprivitere]

For folks finding this who are wondering how to still use kubectx when you have lots of kubeconfig files, here's an example for your .zshrc file to generate a KUBECONFIG variable that kubectx can read.

#Lots of kubeconfig files!
# If there's already a kubeconfig file in ~/.kube/config it will import that too and all the contexts
DEFAULT_KUBECONFIG_FILE="$HOME/.kube/config"
if test -f "${DEFAULT_KUBECONFIG_FILE}"
then
  export KUBECONFIG="$DEFAULT_KUBECONFIG_FILE"
fi
# Your additional kubeconfig files should be inside ~/.kube/ and named kubeconfig.*.yml or kubeconfig.*.yaml
OIFS="$IFS"
IFS=$'\n'
for kubeconfigFile in `find "$HOME/.kube/" -type f -name "kubeconfig.*.yml" -o -name "kubeconfig.*.yaml"`
do
  export KUBECONFIG="$kubeconfigFile:$KUBECONFIG"
done
IFS="$OIFS"

I also find myself using kubie when I need to lock a particular session to a specific kubeconfig context/namespace. It's not as quick as kctx, but the session you set up with kubie will stay on that kubeconfig even if you use kctx to change it everywhere else.

[air3ijai]

For folks finding this who are wondering how to still use kubectx when you have lots of kubeconfig files, here's an example for your .zshrc file to generate a KUBECONFIG variable that kubectx can read.

Your solution does not work for me or maybe I miss something?

1. macOS - zsh / kubectx_v0.9.5_darwin_arm64.tar.gz

   KUBECONFIG=/Users/user/.kube//kubeconfig.cluster-1.yaml:/Users/user/.kube//cluster-2.yaml:
   

2. Linux - bash / kubectx_v0.9.5_linux_x86_64.tar.gz

   KUBECONFIG=/root/.kube/kubeconfig.cluster-1.yaml:/root/.kube/kubeconfig.cluster-2.yaml:
   

Can you please provide an example of the KUBECONFIG variable with multiple files?

[Pinimo]

Easy workaround based on this StackOverflow answer. Write your different configs in multiple files and merge them whenever you need, for instance with the following in your ~/.zshrc (or other shell startup script):

alias kconfig='KUBECONFIG=~/.kube/config.microk8s:~/.kube/config.other kubectl config view --flatten > ~/.kube/config'

This will export a merged configuration to ~/.kube/config live. Note that it will also override the previous configuration, however. Having an alias is useful in my case (one file is dynamic), however it is also possible to use directly this:

KUBECONFIG=~/.kube/config.microk8s:~/.kube/config.other kubectl config view --flatten > ~/.kube/config