SSH driver: --ssh-user argument seems to be ignored

[vrubiolo]

Steps to reproduce the issue: minikube -p vm-cloud-k8s start --driver=ssh --native-ssh=false --ssh-user=vrubiolo --ssh-ip-address=vm-cloud --ssh-key=~/.ssh/id_rsa-vm_cloud --alsologtostderr -v=4 2>&1 | tee minikube.log

We can see futher down that minikube tries to connect using the root user, bypassing the --ssh-user option:

* Updating the running ssh "vm-cloud-k8s" bare metal machine ...
I0930 16:05:21.592604  898636 machine.go:88] provisioning docker machine ...
I0930 16:05:21.592842  898636 main.go:130] libmachine: Waiting for SSH to be available...
I0930 16:05:21.592906  898636 main.go:130] libmachine: Getting to WaitForSSH function...
I0930 16:05:21.593313  898636 main.go:130] libmachine: Using SSH client type: external
I0930 16:05:21.593462  898636 main.go:130] libmachine: &{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@vm-cloud -p 22] /usr/bin/ssh <nil>}
I0930 16:05:21.593526  898636 main.go:130] libmachine: About to run SSH command:
exit 0
I0930 16:05:21.703314  898636 main.go:130] libmachine: SSH cmd err, output: exit status 255: 
I0930 16:05:21.703385  898636 main.go:130] libmachine: Error getting ssh command 'exit 0' : ssh command error:
command : exit 0
err     : exit status 255

The machine does not allow SSH using the root account, which is why this fails. One can see this via the SSH command in verbose mode (this is why I used the external SSH binary):

$ /usr/bin/ssh -F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=verbose -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@vm-cloud -p 22
Warning: Permanently added 'vm-cloud' (ED25519) to the list of known hosts.
root@vm-cloud: Permission denied (publickey).

Workaround Enable root account over SSH (if possible) for provisioning to continue.

Run minikube logs --file=logs.txt and drag and drop the log file into this issue minikube.log

This is minikube v1.23.1 on Fedora 34

[vrubiolo]

Following discussions on #10618 and #10578, I have checked my profile has the right parameters (attached):

 37     "SSHIPAddress": "vm-cloud",
 38     "SSHUser": "vrubiolo",
 39     "SSHKey": "~/.ssh/id_rsa-vm_cloud",
 40     "SSHPort": 22,

config.txt (renamed for filetype compliance w/ Github)

[sharifelgamal]

cc @afbjorklund

[afbjorklund]

The SSHUser parameter looks good, wonder why it is getting replaced with "root" (in the ssh command) ?

EDIT: Seems to be a bug in GetSSHClientFromDriver, but not sure how because it looks legit ?

        client, err := ssh.NewClient(d.GetSSHUsername(), address, port, auth)

func (d *Driver) GetSSHUsername() string {
        return d.SSHUser
}

[afbjorklund]

I think I saw this before, there is a fork of the ssh code that uses the host user instead of the driver user.

It is the same one that hardcodes NewNativeConfig, or it could be related to the bug with the driver registry

cmd/minikube/cmd/start_flags.go: defaultSSHUser = "root"

For some reason, the config gets reset to the default values...

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[vrubiolo]

I am still interested in a fix for this but a workaround exists (enable the machine root account to allow provisioning).

As per the instructions, I am refreshing this.

/remove-lifecycle stale

[vrubiolo]

/remove-lifecycle stale

[sharifelgamal]

I'll freeze this so it doesn't go stale while someone looks into it. Help wanted!

[sgmv]

The same problem. --ssh-user doesn't work correct :(

[kadern0]

I've observed a funny behavior that "could explain" this issue. After testing the ssh driver a few times, I've noticed that if I reuse a profile that hadn't worked on a previous attempt (because of a bad initial configuration), minikube will try to use the default ssh account (hardcoded to root).

This one uses the correct username:

 ./out/minikube start -p testing01 --driver=ssh --native-ssh=false --ssh-user=pcaderno --ssh-ip-address=desktop --ssh-key=~/.ssh/id_rsa-minikube  --alsologtostderr -v=4 2>&1
...
I0929 09:54:51.332191 2845111 main.go:134] libmachine: &{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null pcaderno@desktop -p 22] /usr/bin/ssh <nil>}

This one doesn't:

./out/minikube start -p test --driver=ssh --native-ssh=false --ssh-user=pcaderno --ssh-ip-address=desktop --ssh-key=~/.ssh/id_rsa-minikube  --alsologtostderr -v=4 2>&1

...
I0929 09:55:08.336979 2845144 main.go:134] libmachine: &{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@desktop -p 22] /usr/bin/ssh <nil>}

@vrubiolo , @sgmv could you try to test this theory by running the same minikube command using a new profile name and let me know if you observe the same thing?

[zarinfam]

I have this problem with the --ssh-user parameter.