openssl pkeyutl error Can't set parameter rsa_padding_mode:oaep when decrypting RSA-OAEP ciphertext

[licuser]

Hi, Using OpenSSL 3.0 and Tpm2 Tools version="5.5" I am facing a problem when decrypting an RSA-OAEP encrypted data with SHA1. I got error pkeyutl: Can't set parameter "rsa_padding_mode:oaep": as described below: wrap primary key creation tpm2_createprimary -C o \ -g sha256 \ -G rsa \ -p $PASS \ -c enroll_rsa.ctx HANDLE=$(tpm2_evictcontrol -c enroll_rsa.ctx | cut -d ' ' -f 2 | head -n 1) keypair creation openssl genpkey -provider tpm2 -propquery '?provider=tpm2' \ -algorithm RSA \ -pkeyopt bits:2048 \ -pkeyopt parent:${HANDLE} \ -pkeyopt parent-auth:$PASS \ -pkeyopt user-auth:$USER_PASS \ -out machine.sk.pem Encrypt data openssl pkeyutl -encrypt -inkey machinepubkey.pem -pubin -in msg.txt -out msg.enc -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 decrypt data TPM2OPENSSL_PARENT_AUTH=PARENTPASSPHRASE openssl pkeyutl -provider tpm2 -provider base -propquery '?provider=tpm2' -inkey machine.sk.pem -passin pass:keypassword -decrypt -in msg.enc -out msg2.txt -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha1

PROVIDER INIT DER DECODER DECODE DER DECODER DECODE TSS2 DECODER DECODE 0x87 TSS2 DECODER LOAD parent: persistent 0x81000000 TSS2 DECODER DECODE 0x87 TSS2 DECODER LOAD parent: persistent 0x81000000 TSS2 DECODER DECODE found RSA RSA LOAD RSA GET_PARAMS [ bits security-bits max-size ] RSA HAS 1 DECRYPT INIT DECRYPT SET_CTX_PARAMS [ pad-mode ] pkeyutl: Can't set parameter "rsa_padding_mode:oaep": RSA FREE PROVIDER TEARDOWN

and if I remove padding keyopt from command: TPM2OPENSSL_PARENT_AUTH=PARENTPASSPHRASE openssl pkeyutl -provider tpm2 -provider base -propquery '?provider=tpm2' -inkey /mnt/licpart/enrollement/machine.sk.pem -passin pass:keypassword -decrypt -in msg.enc -out msg2.txt

PROVIDER INIT DER DECODER DECODE DER DECODER DECODE TSS2 DECODER DECODE 0x87 TSS2 DECODER LOAD parent: persistent 0x81000000 TSS2 DECODER DECODE 0x87 TSS2 DECODER LOAD parent: persistent 0x81000000 TSS2 DECODER DECODE found RSA RSA LOAD RSA GET_PARAMS [ bits security-bits max-size ] RSA HAS 1 DECRYPT INIT DECRYPT WARNING:esys:/var/tmp/portage/app-crypt/tpm2-tss-4.0.1/work/tpm2-tss-4.0.1/src/tss2-esys/api/Esys_RSA_Decrypt.c:305:Esys_RSA_Decrypt_Finish() Received TPM Error ERROR:esys:/var/tmp/portage/app-crypt/tpm2-tss-4.0.1/work/tpm2-tss-4.0.1/src/tss2-esys/api/Esys_RSA_Decrypt.c:102:Esys_RSA_Decrypt() Esys Finish ErrorCode (0x00000084) Public Key operation error 40F76343C77F0000:error:40000012:tpm2:decrypt_message:cannot decrypt:src/tpm2-provider-asymcipher-rsa.c:81:132 tpm:handle(unk):value is out of range or is not correct for the context RSA FREE PROVIDER TEARDOWN

Any help please?

[JuergenReppSIT]

@gotthardp Any idea?

[gotthardp]

Yeah, the OAEP support is not implemented, see https://github.com/tpm2-software/tpm2-openssl/issues/89.

[gotthardp]

I just added support for the OAEP padding, so if you build the latest tpm2-openssl (master branch), your script may work.

[licuser]

Perfect, it works fine now! Thank you