Update S-exp parsing code to maintain compatibility with newer GnuPG versions.

[ni4]

Description

Looks like after some update (didn't dig into this yet) GnuPG switched to text representation of the S-exps instead of previously used binary ones. We should update our code to support all possible representations for future compatibility. See more on S-exps here: https://people.csail.mit.edu/rivest/Sexp.txt And GnuPG's implementation here: https://lists.gnupg.org/pipermail/gnupg-devel/2019-July/034438.html This links could be useful: https://people.csail.mit.edu/rivest/sexp.html

Command sexp-conv should be useful.

[ni4]

This was changed since GnuPG 2.3+ : https://github.com/gpg/gnupg/blob/master/agent/keyformat.txt

[ronaldtse]

Let's handle SEXP in a separate reusable library: https://github.com/rnpgp/sexp/issues/1

[maxirmx]

UPD GnuPG 2.3+ not only changes SEXP encoding from canonical to advanced format but also applies openpgp-s2k3-ocb-aes protection for private key instead of openpgp-s2k3-sha1-aes-cbc Although openpgp-s2k3-ocb-aes is supported by rnp in the large, it is applied to key file decoding incorrectly. I am trying to fix it although GnuPG documentation at https://github.com/gpg/gnupg/blob/master/agent/keyformat.txt does not look 100% reliable that creates additional problems.

[maxirmx]

UPD GnuPG 2.3

• saves key using advanced transport SEXP encoding instead of canonical encoding
• encapsulates key into name-value file structure
• applies openpgp-s2k3-ocb-aes protection for private key instead of openpgp-s2k3-sha1-aes-cbc
• uses unprotected sections of S-expression as additional authenticated data for AEAD block cipher that is used to encrypt private key. Furthermore, these unprotected sections are passed to gcry_cipher_authenticate in cannonical encoding (and not advanced transport)

[maxirmx]

UPD It is implemented for Botan backend. Support of GnuPG extended key requires AEAD_OCB Cipher and I was not able to setup it in compatible mode It is related to https://github.com/rnpgp/rnp/issues/1642 but does not require full support.
OpenSSL Cipher passes tests provided at RFC 7253(Appendix A) but fails to decrypt GnuPG protected key

[ronaldtse]

@maxirmx the RNP OpenSSL implementation does not yet support AEAD yet and is being implemented by @ni4 . Maybe we should only release this once AEAD is done?

[maxirmx]

@ronaldtse
We have three options:

1. Release support of GnuPG extended key format for botan backend only (openssl backend will return an error)
2. Implement limited support of AEAD for openssl backend for now - only the features that are required for GnuPG private key unprotection. RFC 7253 OCB Cipher tests with AD are passing for openssl backend, so it is somewhere close.
3. Postpone support of GnuPG extended key format until AEAD support is fully implemented for openssl backend.

My choice would be (2)

[ronaldtse]

@maxirmx I agree that the most reasonable option is (2). Any thoughts @ni4 @antonsviridenko ?

[antonsviridenko]

yeah, option 2 seems reasonable

[ni4]

@ronaldtse Sorry for the delayed reply. Agree with (2), given that OpenSSL provides only AES-OCB implementation.