TLS issue with cert in TPM - works in OpenSSL 1.1
So we are using openssl to have a TLS server with the private key in the TPM. we are able to access it with Openssl 1.1 using tpm2tss engine and are able to successfully use the server and client as below
openssl s_server -cert ssl_certificate.pem -key 0x81000008 -keyform engine -engine tpm2tss -cipher AES128-GCM-SHA256 -debug -msg -accept 1234 -cert_chain combinedchain.pem -min_protocol TLSv1.2 -max_protocol TLSv1.2
And the client we used
openssl s_client -connect localhost:1234 -cipher AES128-GCM-SHA256 -debug -msg -min_protocol TLSv1.2 -max_protocol TLSv1.2
In OpenSSL3, we tried the tpm2 provider and we are getting the TLS alert decrypt error during the TLS handshake (with the same client connection as above)
openssl s_server -provider tpm2 -provider default -propquery '?provider=tpm2' -accept 1234 -key 0x81000008 -cert ssl_certificate.pem -cert_chain combinedchain.pem
This is the error in Openssl3 server
LS 1.2, Alert [length 0002], fatal decrypt_error
02 33
ERROR
40F7D51F207F0000:error:0A00041B:SSL routines:ssl3_read_bytes:tlsv1 alert decrypt error:../ssl/record/rec_layer_s3.c:1584:SSL alert number 51
shutting down SSL
CONNECTION CLOSED
The below is the error in Openssl3 client
write to 0x560af1908c10 [0x560af1919080] (7 bytes => 7 (0x7))
0000 - 15 03 03 00 02 02 33 ......3
TLS 1.2, Alert [length 0002], fatal decrypt_error
02 33
4017B53BF87F0000:error:02000088:rsa routines:RSA_verify_PKCS1_PSS_mgf1:salt length check failed:../crypto/rsa/rsa_pss.c:116:expected: 32 retrieved: 222
4017B53BF87F0000:error:1C880004:Provider routines:rsa_verify:RSA lib:../providers/implementations/signature/rsa_sig.c:815:
4017B53BF87F0000:error:0A00007B:SSL routines:tls_process_key_exchange:bad signature:../ssl/statem/statem_clnt.c:2299:
So we are not sure how to pass any additional parameters to pass(so we dont get salt length check failure) in openssl 3 with tpm2 provider so that we dont get the salt length error. the TPM could not be the issue as it works perfectly for the same keys in openssl1 on the same machine ?