"invalid skaffold config" error with multi-node minikube cluster

[fengye87]

trafficstars

1. create a multi-node minikube cluster using minikube start --nodes 2
2. run skaffold dev

Expected behavior

It should work as expected

Actual behavior

It failed with following message:

invalid skaffold config: getting minikube env: running [/usr/local/bin/minikube docker-env --shell none -p minikube]
 - stdout: "\n\n"
 - stderr: "X Exiting due to ENV_MULTINODE_CONFLICT: The docker-env command is incompatible with multi-node clusters. Use the 'registry' add-on: https://minikube.sigs.k8s.io/docs/handbook/registry/\n"
 - cause: exit status 81

Information

• Skaffold version: v1.20.0
• Operating system: CentOS 7
• Installed via: skaffold.dev

Steps to reproduce the behavior

1. a clonable repository with the sample skaffold project
2. minikube start --nodes 2
3. skaffold dev

[fengye87]

The cause of this problem is that minikube docker-env is not supported with multi-node clusters. So we may need a way (a flag maybe?) to let skaffold treat multi-node minikube clusters as non-minikube clusters, and rely on a Docker registry for the build-push-deploy process just like on canonical k8s clusters.

[briandealwis]

And --detect-minikube=false doesn't work if the cluster is named minikube. We have some rework to do with Minikube's support for containerd too (#5143), and here's another item to add to the list.

As a workaround you should be able to run:

skaffold config set -k minikube local-cluster false

[fengye87]

@briandealwis The workaround you mentioned didn't seem to solve the problem. I still got the same error.

[MarlonGamez]

hey @fengye87, unfortunately I don't think we have a workaround for this right now, however, we are actively working on support for minikube's new minikube image load command which should allow us to no longer rely on minikube docker-env and therefore help us support multi-node minikube

cc @tstromberg @medyagh

[MarlonGamez]

Also for more context, this isn't even happening in the build phase, but rather when we parse the skaffold config. It looks like we try to run validations against the user's docker daemon in minikube at the beginning of a skaffold session

[aasmall]

Interesting to note, this applies even if you use the kaniko builder, which shouldn't require access to the minikube docker daemon. I will close #5702 as a duplicate, but there is more information there, as well as the local patch I used to work around the problem.

[afbjorklund]

You get the same bug when using other container runtimes, since the docker daemon is not running in the cluster for those. Even if you are using a custom build script (without docker), it will still try to call minikube docker-env and fail while doing so.

[tejal29]

We would love get some community contributions arounds this!

[rmane-vibrent]

try doing:: local: push: false

it helped me

[gawsoftpl]

Still not working --detect-minikube=false dont resolve this issue

invalid skaffold config: getting minikube env: running [/usr/local/bin/minikube docker-env --shell none -p api-server --user=skaffold]
 - stdout: "false exit code 81\n"
 - stderr: "X Exiting due to ENV_MULTINODE_CONFLICT: The docker-env command is incompatible with multi-node clusters. Use the 'registry' add-on: https://minikube.sigs.k8s.io/docs/handbook/registry/\n"
 - cause: exit status 81

[Jonatthu]

This is working with KIND cluster BTW

[yuhang-lin]

@rmane-vibrent I've tried local: push with both true and false, but none of them worked.

[gawsoftpl]

I move from minikube to kind.

[Woytek-Polnik]

If you rename context from "minikube" to for example "test" and tell skaffold to use SKAFFOLD_KUBE_CONTEXT=test it works....