sops-nix Some SSH RSA keys cannot be used for decryption...

I'm trying this out for the first time, deploying just a single secret to a single host with deploy-rs, but I'm getting the following error:

⭐ ❓ [activate] [DEBUG] Running activation script
updating GRUB 2 menu...
activating the configuration...
setting up /etc...
setting up secrets...
sops-install-secrets: Imported /etc/ssh/ssh_host_rsa_key with fingerprint e2083f651825666614144c58e6d961dca7ddf0c4
Activation script snippet 'setup-secrets' failed (1)
/nix/store/n6b7y2hj7ymd11xnigkhpyysx3zk83jn-sops-install-secrets-0.0.1/bin/sops-install-secrets: Failed to decrypt '/nix/store/iv09rddx0hfs6bk6262vacwly3g28cw8-secrets.yaml': Error getting data key: 0 successful groups required, got 0
reloading user units for root...
reloading user units for dhess...
setting up tmpfiles
⭐ ⚠️ [activate] [WARN] De-activating due to error

The machine GPG key was imported from its SSH host RSA key as explained in the README. The other key I'm using to encrypt this particular secret is my own personal GPG key. I can edit (decrypt) and encrypt the secrets.yaml file on my local machine fine.

On the server, when I try to run the sops-install-secrets command on the manifest, I get the same error as above:

/nix/store/n6b7y2hj7ymd11xnigkhpyysx3zk83jn-sops-install-secrets-0.0.1/bin/sops-install-secrets /nix/store/mzs4sqzkawwp995jv8svf247rwzpz3iy-checked-manifest.json
sops-install-secrets: Imported /etc/ssh/ssh_host_rsa_key with fingerprint e2083f651825666614144c58e6d961dca7ddf0c4
/nix/store/n6b7y2hj7ymd11xnigkhpyysx3zk83jn-sops-install-secrets-0.0.1/bin/sops-install-secrets: Failed to decrypt '/nix/store/iv09rddx0hfs6bk6262vacwly3g28cw8-secrets.yaml': Error getting data key: 0 successful groups required, got 0

Here are the secrets.yaml file and the manifest:

https://gist.github.com/dhess/74ec4eaa364369420e4121fa8ecbdd92 https://gist.github.com/dhess/a792710fbc263f899f75b2b3879e3e71

Jan 25 '21 01:01 dhess

That look indeed weird. You imported key e2083f651825666614144c58e6d961dca7ddf0c4 also is used to in the sops file. So you should have encrypted the manifest correctly as far as I can tell.

Jan 25 '21 09:01 Mic92

Per your advice on IRC, I tried this with a different SSH key that I generated specifically to test this (i.e., this is not a production SSH RSA host key). On the remote host, I put the key in /var/lib/sops/ssh_host_rsa_key and added this NixOS config:

        sops.sshKeyPaths = [
          "/var/lib/sops/ssh_host_rsa_key"
        ];
        sops.secrets.radius-cert-key = {
          mode = "0400";
        };

Here's an example secrets file. On my setup, this file fails to decrypt properly when deployed to the host with the above config.

radius-cert-key: ENC[AES256_GCM,data:U3QQdxvDtT6Rzfv6qtYr,iv:ef1e1tgSI+4nMJeoriwHzLfT9qwcG3XdFshoTVPHQ48=,tag:OiOqXpC06mUGkHQwR4HFyQ==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    lastmodified: '2021-01-25T10:12:55Z'
    mac: ENC[AES256_GCM,data:ouVwHl22XlVYpkks1yxBqSChfx8FHeysmQeNCXXaIdNcjaH+TKOZGJbo6g+nUFXXWa7XeH52+bfuebfJjUVw1M1ejI8Iar26BlnTuQRtjXzqCiNZ0HVEfhsNi7rSBVd7TPXDzBNNggkemYB6zB2BQ0pIoZqBA06y06b/2XSzrBE=,iv:SYeXocc9rInhVrN79od4vawpqkblKiSAf1U/UsvQA4w=,tag:f47hNScD0qzwSSFXffUYLQ==,type:str]
    pgp:
    -   created_at: '2021-01-25T00:09:09Z'
        enc: |
            -----BEGIN PGP MESSAGE-----

            hQIMAwAAAAAAAAAAAQ/+MHspu2C+TWqc/t/wU8TOp2SatspnQsfD4krI0aIO39Z8
            smrmIkAz9ECpcFDj1r1WqXcv90DyWtOsMZTfOaFCQiEG0mSePYy0oSMHah15/la8
            NAFbv20sn0KUaV0R++lism+P3QClWZmzPx8DINGVBaPsV05fhc4CAGfvCusGF1+b
            P8AkHSGnAa5oK7DDDWUa8d1f4xN3HUy9WM1A2/UaxHv6M4WSwLpBI/jStRQyszMh
            jnItzThSxr4mK96Z5/Sc8pd4ZU78T9xqEh/r+bRs07LZCUfnwok9rE4uLtuvxU+b
            wKP21Iuq5VDRT7Ba8wrVS8kP0b/eFyje2ZaeTr0ivnoQLDAVShP8fRAzeaeGLmsb
            QSQZ0vUe2oOUuCbGpL0aNjUwumvkzJC2QZ1MGK8aj6Dlry4DGaouuXq+AWCZsp9C
            GApfop/bPzuMKY44BMB6ZWcPdIOxHV5h1pb5hHHtoz7cbc/X0tphPlOgyai29Cvm
            aQ65HIN/9UqZF5VbWC/+zT1YWJ1m5/dFUfWKJaNya6xQvPlxplOdgVeMM7lQW3/z
            g8N1US9LDhjZbB1dPbRCtUeeQKLvReTfFUJmVE99P8Ji96NU9kTKVCY+ZkgUCOrl
            zUU3i9Dmd7q61wk4yAlIdGvn/yjj6gIClZvYd+0JZlepbPiXlJ70LGE66tNnZB7S
            XAGuAlk3EVFiHVk07un7Otxfk4mOxiwFJMbIosY0uu7Fpk/pe4kB5X7Iv4/rFvUv
            Y3EPBZjD7WUq1Ej1dC2V8HFQN0XktVW3JaXZ/tWSvZEHc/UP5hj7S/DzFSzt
            =KQPj
            -----END PGP MESSAGE-----
        fp: F2B90CD1A9634E3DBF29AE6C2A20D2096BABEC61
    -   created_at: '2021-01-25T00:09:09Z'
        enc: |
            -----BEGIN PGP MESSAGE-----

            hQIMAwAAAAAAAAAAAQ//e64GMXNVa4XdJSDsddZsjA1PHOpDEbMIuzPvstZ01fPl
            aXs83q2Nw1BzYGIeEAtRpTVIAWSy/tvVSPhcbqf/2UUhd3V9OGXiMRRpoz0lIG/R
            +4afSnDHrA2F7x8SAPoECjmIRwDlZWpFFSOnezHaSi4z/aOei2bu2tTpaxRigUCq
            MqdMifxJSGxEHopRP7CnvXJsxIvxXahZsva4zfNcmoGCWMUuOR57ZQK5Da4Sxtdb
            7M9x+rMhUfpyCfwbBXuKUVVz0pVxTdCXNFzMTnevqfjAWS0RRTJtrIEZeheE4gtn
            Schzb+0H0YoO4qGMOiKhmSRpSME01pTQ6824D+Q9UXfZWLwS/+JQ8YNJYq0r6Qx4
            IZ8TLAuttcB/78cPo1YKf+wNvGGSMt4o/FpJvrrn3EMl9zRugaqrEAz9Kv2xy8Cg
            jrYfLKncIu4hvwYOdqE2tCMXEh74w6Kw+GcDmJ2aXNcm0Ia7cmZ3RmuByznx2wRm
            WFfF5TBA684T6dCbYzGjJSvEeCrb6ReiPNVrkCc947QAq6G1m/pqBa6OcqIvHIqC
            vbB2MJlEyHe7JjbDAjevhwOG4b+WMLO5Q6/3Tn3WH5zejdtqjPN6kGDXdcYl0KlU
            f/T7IIbFI/1W3OSKg8tjNMfYanKAifta9eZK8+E9RG1qWhEHZw0sDQJLCb7sChbS
            TgHVI0HmnmIgxTimXeDYsjxkBeQLPwCLwYE5dBj/NBvuJhUiqKoxsCCGA09gv+io
            4xMLQr6yXmSXlxCuMhe/mOet0JVz7RFCf1qJtsTkGw==
            =TSxS
            -----END PGP MESSAGE-----
        fp: E2083F651825666614144C58E6D961DCA7DDF0C4
    -   created_at: '2021-01-25T10:03:18Z'
        enc: |
            -----BEGIN PGP MESSAGE-----

            hQGMAwAAAAAAAAAAAQv+MELqWlvm+d2EE+VdBn1aGOmEAZCMI77xGt3Oc84Ucbrz
            WfJ1jaOeTOAeryyfvyrJnPtJNtfeEQWQbueb3Ffdogvwwfz3TRTjjC3UY051TkD0
            iMPpTz5scvfZMMptfet+GIt2cVldsOQXj5OL/howc5sukSpmomEs06lFnfGZ7bnl
            WQneUtEZoVGrvnU7Bi8EvgzYHSQPwtv8ifSJXQLgXWtxXtPBoO//Wd1Hb2okG95j
            TAxrxgUoQc2vL0fzpo0sFXJij6JFfxOkdObVokxnN0E0B3mrwt7ZivTGwbINQ68G
            qFm8mlHVNPzWIpYq3nsK3SohE6mbUusBKDWLiHksqjIrqM26SGPX06hKUZgrX9Mz
            ogrDfHx7jyt6S7g4UmLIxryuWJzlHBTkC46ezlQY75DrvGeq5LcrO0w3GF4JiM7U
            ZyDY/xLYyz2ojtax3NO56gGB+oB6upiTQ5SFTMAz3vum3hysOFA1kN6NuOrZsI6m
            1Kc1MA3VKcSZpAmJ4PLR0k4BcCSEfrCgk3iRqlCdo+s8dP2DMW3A0PI2zGWLSxRb
            dtXLviLAX5Ug7rT3BOPzbRuOGPV6umXy1Tz8oCoEnmij0q5iKSt8rjMorq7g3lI=
            =itHK
            -----END PGP MESSAGE-----
        fp: A58A4F96E0DC4BB0A1AA04D0E845F766451D48EC
    unencrypted_suffix: _unencrypted
    version: 3.6.1

It's encrypted with my personal GPG key, the server's actual SSH host RSA key, and the server's test SSH host RSA key (A58A4F96E0DC4BB0A1AA04D0E845F766451D48EC).

Here is the test SSH host RSA key:

And here, just for reference, is the PGP public key I get when I convert it to a PGP key using ssh-to-pgp on my local system:

Jan 25 '21 10:01 dhess

I was able to reproduce your bug here: https://github.com/Mic92/sops-nix/pull/70

Jan 27 '21 06:01 Mic92

Does #70 also contain a fix? (It's unclear.) And if so, what was the issue?

edit It appears not :)

Jan 27 '21 09:01 dhess

@Mic92 @dhess I just stumbled upon the same issue and I'm also trying to decrypt a file using the remote server's hostkey during activation. Did anyone of you already find a workaround for the problem?

Feb 02 '21 22:02 Ma27

I have not. :(

Feb 02 '21 22:02 dhess

@Mic92 @dhess I just stumbled upon the same issue and I'm also trying to decrypt a file using the remote server's hostkey during activation. Did anyone of you already find a workaround for the problem?

Is it maybe the key length? I noticed that my host ssh key is 4096 bits. This ssh key is 3072. @Ma27 what is your key length where it did not work?

Feb 02 '21 22:02 Mic92

$ ssh-keygen -lf /etc/ssh/ssh_host_rsa_key
4096 SHA256:K7LpWzOS8LD0TQpg5MsfIaH70GsB6xF9NMZhC4abpqk root@turingmachine (RSA)

Feb 02 '21 22:02 Mic92

@Mic92 whoops, I used the wrong tool in IRC for the length, now the actual output:

[root@roflmayr:~]# ssh-keygen -lf /etc/ssh/ssh_host_rsa_key
4096 SHA256:uwFkfH0eHK1P3DVUincD4pv2WX2CZzXrF9j5IL60TkE root@hetzner-ax41-nvme-provisioned (RSA)

Feb 02 '21 22:02 Ma27

Mhm. I need to debug the crypto libraries further to see where this issue is coming from. But at least I have a reproducing key.

Feb 02 '21 22:02 Mic92

Also when I tried to use gpg directly when decrypting the sops keys it worked. So it might be a bug in golang's gpg library.

Feb 03 '21 05:02 Mic92

Hmm, so my problem smells pretty much like pebcac due to a sleepy Ma27 I'm afraid.

After reading an old excerpt from #nixos-chat I forcefully recreated the secrets file and everything was fine.

There might be a slim chance that this is an actual bug, but IMHO it's far more likely that I wrongly re-encrypted the file in question after I added the key for my server and thus the decrypt failed because the secret wasn't encrypted for the server's host key.

In that case, sorry for the noise and thanks a lot for this awesome sops integration!

Feb 03 '21 09:02 Ma27

@Mic92 @dhess I just stumbled upon the same issue and I'm also trying to decrypt a file using the remote server's hostkey during activation. Did anyone of you already find a workaround for the problem?

Is it maybe the key length? I noticed that my host ssh key is 4096 bits. This ssh key is 3072. @Ma27 what is your key length where it did not work?

FYI, the original host SSH key I used, which also does not work, is 4096 bits.

Feb 03 '21 09:02 dhess

I find this issue really puzzling. I've now been able to reproduce the problem across multiple projects, on multiple hosts, using multiple methods of building and deploying NixOS hosts. One project uses Nix Flakes' nixosConfiguration method of building systems and deploys them using deploy-rs, and the other uses standard Nix (without Flakes) and NixOps.

In every single case, on every single host, I get this error when I try to use the host's SSH key to encrypt the secrets. But with the same configurations and secrets, if I create a GPG key for each host using the method given in the README, everything works fine. I must be missing something really fundamental, because otherwise, I can't imagine how anyone gets the SSH-derived keys to work :)

Feb 09 '21 15:02 dhess

I find this issue really puzzling. I've now been able to reproduce the problem across multiple projects, on multiple hosts, using multiple methods of building and deploying NixOS hosts. One project uses Nix Flakes' nixosConfiguration method of building systems and deploys them using deploy-rs, and the other uses standard Nix (without Flakes) and NixOps.

In every single case, on every single host, I get this error when I try to use the host's SSH key to encrypt the secrets. But with the same configurations and secrets, if I create a GPG key for each host using the method given in the README, everything works fine. I must be missing something really fundamental, because otherwise, I can't imagine how anyone gets the SSH-derived keys to work :)

Sorry I have not yet debugged the ssh key yet to the very heart of the go crypto library, but you must be somehow special since you are the only person I am aware of that can build invalid ssh keys.

Feb 09 '21 16:02 Mic92

I encountered the same issue while trying sops-nix. I'm also using Nix Flakes for my project and deploy-rs as deploy tool. When using GnuPG keys instead of ssh keys everything works as expected.

Aug 31 '21 08:08 welteki

This issue continues to plague me, even on new installations with and without flakes, with both deploy-rs and nixops. I have never once gotten the SSH-derived key to work.

Aug 31 '21 09:08 dhess

maybe SOPS_GPG_EXEC needs to be set by default: https://github.com/Mic92/sops-nix/blob/3e4ebc851c91d1ce5c65da23436726c555a0d7e8/modules/sops/default.nix#L173

Aug 31 '21 14:08 Mic92

I may make this change once we also have age support: https://github.com/Mic92/sops-nix/pull/107 I suspect age will be much more predictable.

Aug 31 '21 14:08 Mic92

Age support is now available.

Sep 24 '21 12:09 Mic92

@Mic92: I just tried age support. It seems I need to set sops.gnupg.sshKeyPaths to an empty list for age keys to work. In other words, this won't work:

{
  sops = {
    defaultSopsFile = ./secrets.yaml;
    age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
    # gnupg.sshKeyPaths = [];
    secrets.my-secret = {};
  };
}

Failure:

activating the configuration...
setting up /etc...
setting up secrets...
sops-install-secrets: Imported /etc/ssh/ssh_host_rsa_key with fingerprint 0cb4718909edfa926f70602b810663e262ae61eb
/nix/store/z3qzgbhfc3wvs9si9hch4n2zdvvbhz36-sops-install-secrets-0.0.1/bin/sops-install-secrets: Failed to decrypt '/nix/store/h4wzraa8br37wxryya3v98wjh4x93dc4-secrets.yaml': Error getting data key: 0 successful groups required, got 0
Activation script snippet 'setup-secrets' failed (1)
reloading user units for jani...

While this works:

{
  sops = {
    defaultSopsFile = ./secrets.yaml;
    age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
    gnupg.sshKeyPaths = [];
    secrets.my-secret = {};
  };
}

activating the configuration...
setting up /etc...
setting up secrets...
reloading user units for jani...

Notice how it tries to use ssh_host_rsa_key instead of ssh_host_ed25519_key. It seems if an RSA keys is found, ED25519 keys aren't tried.

Sep 28 '21 11:09 ljani

I am testing a fix regarding that: https://github.com/Mic92/sops-nix/pull/117

Sep 28 '21 12:09 Mic92

I'm still getting the error if I set keyFile, e.g.:

sops.age.keyFile = "/var/lib/sops-nix/key.txt";

Sep 29 '21 23:09 jlesquembre

@jlesquembre Try setting sops.age.sshKeyPaths = []. Does that fix that?

Sep 29 '21 23:09 dasJ

@dasJ yes, that seems to fix the issue

Sep 29 '21 23:09 jlesquembre

I added assertions here to make this more clear: https://github.com/Mic92/sops-nix/pull/121

Sep 30 '21 05:09 Mic92

Seems there is no fix yet, right?

Oct 27 '21 03:10 twink0r

@twink0r #122 fixed the issue for me

Oct 27 '21 05:10 jlesquembre

@jlesquembre i still face this issue with RSA Keys. I don't know howto debug this.

Oct 27 '21 13:10 twink0r

Can you try switching to ed25519 ssh keys? I don't plan to invest much time into ssh2gpg code base of sops beyond basic maintenance because also the go implementation for pgp has no maintainer anymore and sops also recommend to switch to age.

Oct 27 '21 18:10 Mic92

sops-nix sops-nix copied to clipboard

Some SSH RSA keys cannot be used for decryption...

sops-nix
sops-nix copied to clipboard