Allow extra options to be passed to `ssh` calls.

[soupglasses]

Is your feature request related to a problem? Please describe.

Ssh configurations can be custom defined, when IdentitiesOnly enabled, ssh will no longer iterate over its ~/.ssh/ keys. Instead of requiring each user to configure their global ~/.ssh/config per install, i would propose an extra option, --ssh-option which would allow you to set arguments that are passed into each ssh call, similar to how nix has it with --option.

Describe the solution you'd like

This can help greatly by allowing a pathway to manually define an ssh key.

For example: --ssh-option '-i ~/.ssh/key'.

Describe alternatives you've considered

Manually writing/changing each machine's .ssh/config to make it work with nixos-anywhere. But this seems less than ideal and requires a lot of manual intervention unless ssh defaults are strictly followed by everyone.

Additional context

None

[phaer]

Thanks, I agree that there might be possible improvements in the handling of SSH options, but it's a bit tricky. Nixos-anywhere currently implicitly generates it's own temporary SSH key to use during installation. So for --ssh-option we would need to decide whether to apply those options to all ssh commands or only for the initial connection.

For the specific use case you've mentioned, SSH_PRIVATE_KEY in https://github.com/numtide/nixos-anywhere/blob/193a88b21a994df3d58d83348259514bb319c44b/src/nixos-anywhere.sh#L187 might be sufficient?

I am myself not too excited about the temp ssh key but others here argued that it's necessary to support installation to systems where there isn't a authorized key yet. We could just leave copying pub keys to the target before running nixos-anywhere to the user IMO, but we currently do not.

[soupglasses]

I do not have many smart words on the use of temporary SSH keys. However the SSH_PRIVATE_KEY to get the initial connection is exactly what i was hoping for to exist! As the issue was opened, and i should have put it in additional context, because Hetzner likes you to provision new servers with SSH keys. Making a key the only way to get into the newly made VM.

However, attempting to use SSH_PRIVATE_KEY i keep getting hit by a Load key "/tmp/tmp.xxxxxxxxx": error in libcrypto. Giving it a path, full-path or piped in the file contents of the private file. This may be due to the ssh key being password encrypted?

[soupglasses]

Okay from some diff checking between the /tmp key and the ~/.ssh key. The /diff key gets its newline stripped.

< -----END OPENSSH PRIVATE KEY-----
\ No newline at end of file
---
> -----END OPENSSH PRIVATE KEY-----

Manually opening and adding a ending newline to the /tmp key gets it working again, and asking for the passphrase correctly as well. Letting the ssh connection successfully establish!

[soupglasses]

So to make it work today:

SSH_PRIVATE_KEY="$(cat ~/.ssh/priv_key)"$'\n' nixos-anywhere --flake .#host [email protected]

I would propose making SSH_PRIVATE_KEY use the path to the key ~/.ssh/priv_key instead of copying it around like this. As it is only used for the first initial connection from the known key.

[Mic92]

I thought also having it as a path instead... cc @Lassulus who introduced it for https://github.com/numtide/nixos-anywhere/pull/21/files

[Mic92]

Maybe it should become even just a flag rather than a hidden environment variable. I think it was just hard to do this in terraform in combination with cleaning up temporary files.

[Lassulus]

hmm, terraform has no easy way to write a generated ssh key into a path I guess? I don't want to write another wrapper for running nixos-remote from terraform

[zimbatm]

yes, you can even use https://github.com/hashicorp/terraform-provider-tls to generate the SSL key-pair, and then use https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file to write the file to disk.

[phaer]

hmm, terraform has no easy way to write a generated ssh key into a path I guess?

Whats wrong with local_file?

Tbh I also didn't really understand why terraform supposedly needs generated SSH key thing? Just passing a private key path via variable or defaulting to a local ssh agent has worked before in comparable projects. From my POV it just saves ppl who don't get theirs keys deployed by their cloud provider a single ssh-copy-id command while complicating the logic for everyone else :/

[zimbatm]

Terraform is a bit brittle and tends to break down when using user-provided keys in my experience. You have to make sure that all the users have already an ssh-agent started, otherwise, it will fail with a timeout.

[Lassulus]

yes, you can even use hashicorp/terraform-provider-tls to generate the SSL key-pair, and then use registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file to write the file to disk.

ah, ok, not sure how to write to a tmpfile and delete it after the terraform run again though. seems a bit more overhead which we need to do in every module which needs ssh keys? but if that is the preferred way, we could do it that way.

[Lassulus]

hmm, terraform has no easy way to write a generated ssh key into a path I guess?

Whats wrong with local_file?

Tbh I also didn't really understand why terraform supposedly needs generated SSH key thing? Just passing a private key path via variable or defaulting to a local ssh agent has worked before in comparable projects. From my POV it just saves ppl who don't get theirs keys deployed by their cloud provider a single ssh-copy-id command while complicating the logic for everyone else :/

well, usually the private keys are setup in terraform, so the same installtion can be run by different people or another person can debug the machine without exchanging private keys or populating the machine with additional public keys. some terraform modules also just allow a single key for the created machine, so cooperating becomes very tedious if people use their own keys. usually thats only a problem until we have nixos installed though.

[zimbatm]

I see your point. I suggest adding a little wrapper to the terraform module that copies the passed ssh key to a temp-file and deletes it at the end.

#!/usr/bin/env bash
ssh_key=$(mktemp)
trap 'rm -f "$ssh_key"' EXIT
chmod 0600 "$ssh_key"
echo "$SSH_KEY_CONTENT" > "$ssh_key"
nixos-anywhere --ssh-key "$ssh_key"

[phaer]

tldr; generic ssh hard, agent good. IMO, you decide :+1:

Terraform is a bit brittle and tends to break down when using user-provided keys in my experience. You have to make sure that all the users have already an ssh-agent started, otherwise, it will fail with a timeout.

Maybe I've just been lucky here so far :) I personally think some documentation about getting an ssh-agent started and some keys added could be a good solution here. Pros include that can be a quite flexible solution that can support hardware tokens and ancient ciphers were needed, and that it does not require writing a private key to the file system at all.

The current approach of a generated key and a preference for the systems SSH assumes that ssh-keygen can generate ed25519 keys. That should be true for any non-ancient ssh - but might be a good example that it's hard to write a really portable solution - so just giving up and leave it to users and their ssh-agent seems attractive to me :stuck_out_tongue_winking_eye:

well, usually the private keys are setup in terraform, so the same installation can be run by different people or another person can debug the machine without exchanging private keys or populating the machine with additional public keys. some terraform modules also just allow a single key for the created machine, so cooperating becomes very tedious if people use their own keys. usually thats only a problem until we have nixos installed though.

Thanks for the explanation! As I understand it, we are only talking about the installation process here and the installed system isn't affected by this key as all. A single node installation should be an atomic step from terraforms perspective right? Meaning it either succeeds and all people in a team can log in (public keys in installed system closure or --extra-files).

...or it fails and the node and the node can be destroyed and re-created to see if the error is reproducible. So the only remaining use case I see is shared remote debugging of a "flaky" installs which only fail sometimes. There a various solutions outside the scope of nixos-anywhere and terraform for this (i.e. physical pair programming, a shared tmux session or an "debug-installation" ssh key in SOPS or so). For problematic platforms/machines, it might also make sense to debug the install without terraform, only nixos-anywhere first.