How to rotate keys?

[3noch]

Is it possible to rotate the keys that nixops has generated automatically? At the most basic level this would include the SSH keys for root on each machine.

[spinus]

@3noch I don't think there is any way. I assume you could write a daemon which changes the keys in your deployment host and then it reruns "nixops deploy" to redistribute the keys. But I'm not sure this is way to go. Probably better way would be to implement something like hashicorp vault or some other dynamic secret management.

[3noch]

@spinus Thanks for the idea.

I suppose if I can just add/change the authorized keys for root on the server, then I could actually manually update the deployment data if I had to (via export/import probably). Is the server's root SSH key just stored in /root/.ssh/authorized_keys?

[spinus]

@3noch depends what keys you want to rotate. There I two type of keys in nixops I think. SSH keys which are used to connect to the machine (I'm not sure were are they stored on target machine, probably in some symlinked authorized_keys as you mentioned, but those are only public keys), private keys I think they are in sqlite or in ssh-agent. Second type of keys is user keys declared with "deployment.keys." which are declared in nix expression (or can be read from a file or environment variable) and are deployed to /run/keys/

[3noch]

I've come to the conclusion that some story for this should be very high priority. I recently had a really tough time removing access to a server and had to build a new server from scratch! While that's not nearly as hard to do with NixOps as it would be with some other system, it was still much harder and much slower than it ought to be. If you need to rotate keys, then you likely need to do it quickly.

[ryantm]

@3noch If your setup allows you to remove the statefile for a deployment, you can recreate it, which will effectively rotate the root ssh key.

[3noch]

@ryantm Interesting! How does that work? How would the deployment ssh into the server and change the keys if it didn't have the old ones?

[3noch]

How would it even know the IP address of the server?

[ryantm]

You could copy the old keys somewhere locally, or use some other key temporarily. You can configure additional root keys with users.extraUsers.root.openssh.authorizedKeys.keys. The IP address can be set with deployment.targetHost.

[3noch]

@ryantm Wow ok I'll have to do a trial run of this and see how it goes. This would certainly be a very welcome technique if it indeed works.

[ip1981]

I did this way:

1. Create a new key (with ssh-keygen).
2. Login (as root) to the host.
3. Inspect /etc/ssh/sshd_config to find the place where your root keys rest. Note that different backends may put it in different places. For example, Hetzner - in /etc/ssh/authorized_keys.d/root; AWS - /root/.ssh/authorized_keys. VirtualBox - other fancy place :)
4. Replace the key on the host. (Or add one more, if paranoid, test, replace).
5. Replace the private key in your state. I keep the state in JSON, replacing is simple.

[datafoo]

I was wondering if I could rotate the ssh keys if a team member that had access to the deployment server would leave my company. I ended up here.

It would be a great feature to have indeed.

[jezen]

The instructions above for rolling root SSH keys were super helpful. I found a couple of extra things worth mentioning:

• It's easier to export the deployment, edit the keys in the exported JSON, delete the deployment from the state file, and then import the edited JSON, than it is to edit the state file with sqlite3 directly.
• You may need to do an occasional killall ssh-agent or otherwise kill some running ssh-agent processes for NixOps to use the new key, though this depends on your setup.