Salt length check fails during handshake on Windows with tpm2 on server side (on some TPM modules)

[philippun1]

Hi,

if I try to perform a TLS handshake with tpm2 provider being used on the server side on Windows, I get the following error:

SSL_connect error: error:02000088:rsa routines::salt length check failed

This command is being used to create the certificate:

openssl req -provider-path . -provider tpm2 -provider default -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -sha256 -days 365

The actual error happens in the rsa_pss.c in the function RSA_verify_PKCS1_PSS_mgf1. The Autos tab shows the mismatching sizes:

Although the error happens inside of OpenSSL code, I assume the error is tpm2 provider related. Everything works as expected if I do not load the provider and use a regular certificate.

If it is of any help, I can also test the code on a Linux VM and see if it works there.

Any help is appreciated, even if it is only a hint on where to look at so I can debug into this myself. Thanks.

[gotthardp]

Hmmm. There may be something wrong in the public key or the certificate. Would you please be able to compare the certificate used when it works vs when it doesn't work (openssl x509 -noout -text -in cert.pem)? There may be some metadata missing from the ASN.1 structure that confuses the peer openssl.

[philippun1]

What makes me wonder is the fact that the RSA-PSS code seems to be used, the certificate is RSA though. Might this be the problem?

Here is the output for the certificates I am using.

tpm2 certificate:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            36:ab:c8:67:e8:32:77:99:cb:c5:64:16:c3:5f:3f:37:4f:91:d9:0f
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
        Validity
            Not Before: Jun  7 14:07:28 2023 GMT
            Not After : Jun  6 14:07:28 2024 GMT
        Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:b3:89:9b:75:3a:65:8d:15:e5:a7:ec:4b:64:0c:
                    57:7c:ed:bd:f5:2b:0a:e6:f3:5e:6c:2f:8d:36:e6:
                    be:00:f7:e3:33:dc:a5:22:36:eb:9c:7a:81:e8:10:
                    29:0d:f0:b8:65:78:3c:16:83:8b:65:3a:b2:33:92:
                    f7:ef:97:c3:00:e1:50:f6:a2:3a:bf:8b:9f:b6:37:
                    f7:a1:38:ad:1d:4f:ef:fe:d6:1e:92:cf:85:86:d4:
                    9b:3a:17:52:ec:95:30:39:95:76:54:05:4b:61:31:
                    29:1c:6a:28:6e:6d:2e:51:a3:b3:2d:df:b1:56:4c:
                    55:30:41:4b:29:11:59:60:2a:2e:78:3a:ca:df:7b:
                    cf:c5:50:1b:d3:29:3c:f9:87:ee:ad:0a:ce:47:83:
                    9c:68:82:89:bf:6a:91:e4:3b:dc:f3:6d:a2:56:41:
                    2b:9f:00:bd:9f:0d:8b:98:3c:f4:aa:de:9a:0d:35:
                    ba:9a:28:d6:d5:9d:62:f7:cf:62:75:ea:11:94:21:
                    65:d2:97:36:56:aa:ef:1a:40:28:73:d2:e6:ac:54:
                    a2:4a:ae:7e:e1:58:1e:ac:41:bf:92:ee:99:4b:dd:
                    98:e9:f2:00:4e:84:3e:56:56:07:9a:3f:a7:e2:7d:
                    06:ca:19:1e:31:82:02:54:e6:fd:ad:4a:7d:e8:26:
                    74:1b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                4B:40:56:D1:F7:32:51:E4:FD:05:6B:2B:27:0C:93:D4:1F:FC:70:D4
            X509v3 Authority Key Identifier:
                4B:40:56:D1:F7:32:51:E4:FD:05:6B:2B:27:0C:93:D4:1F:FC:70:D4
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        95:73:b3:b0:32:4d:9d:05:f4:87:ec:9c:2c:02:0b:7a:63:3a:
        82:97:03:a7:ce:4b:b4:c1:2c:a1:55:fc:e4:c2:59:6f:14:4c:
        5f:df:e7:08:6c:be:31:ee:3a:22:6a:39:b6:75:2b:ef:7b:0c:
        64:51:9d:23:d8:ac:d2:a2:d0:6e:3f:cb:28:a2:91:a7:42:ba:
        96:6d:4c:7a:a3:ba:00:eb:5d:03:85:80:41:22:c0:f1:70:0d:
        b8:b8:f9:50:8f:87:95:a4:ac:ad:0f:d2:78:b3:a1:a7:ab:2f:
        36:17:ef:40:70:04:06:e9:d6:dd:02:39:52:ce:31:d7:3b:7b:
        8f:58:ae:de:92:b7:4a:1a:01:f1:2d:03:d3:ea:a8:e1:cd:58:
        eb:fb:76:64:aa:4d:bc:93:23:91:e2:0b:0e:27:44:dc:6e:4c:
        e8:ba:4e:f8:bc:34:e2:c9:41:4a:4b:d0:2e:43:a6:d3:4f:e7:
        91:98:7b:e2:58:16:74:c4:c0:de:0a:80:88:85:3c:32:06:29:
        16:f4:28:2a:c3:de:12:02:64:d6:ec:c4:a0:02:af:55:0a:db:
        ad:49:c9:5d:c2:1d:b0:5e:d7:29:1c:14:96:5c:96:fd:38:b9:
        8c:c8:5a:16:24:0b:ea:b6:20:ec:1d:8c:7a:57:63:72:06:0a:
        fb:fc:46:d0

regular certificate:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            72:f1:c4:2c:a3:e8:45:cc:20:1d:c9:d3:6f:5c:0e:0e:e4:4f:d0:36
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
        Validity
            Not Before: Jun  7 14:08:13 2023 GMT
            Not After : Jun  6 14:08:13 2024 GMT
        Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:89:b8:d3:5c:ef:53:45:dc:e4:4a:ed:8c:fa:c9:
                    2f:34:06:60:06:73:9c:db:62:8d:92:07:dc:19:61:
                    19:5a:02:42:03:b5:61:52:a3:35:67:59:28:0b:ec:
                    d1:91:bb:9c:14:1b:f8:0a:13:29:69:1c:1f:ac:4f:
                    46:4e:d7:e3:38:2b:88:de:10:b1:d2:57:a0:1a:26:
                    5b:12:f5:49:d6:0e:ed:e8:80:a0:d2:d5:55:27:63:
                    43:aa:0a:56:55:71:31:ff:16:11:c1:95:ba:08:1e:
                    e8:6d:79:e9:ed:89:c1:2f:c9:f6:4f:00:9f:e1:7d:
                    08:47:91:c2:b8:24:24:ff:02:5d:9a:08:04:e5:45:
                    32:d7:24:73:46:33:d3:8a:e1:eb:f7:34:3c:4b:e6:
                    8f:74:01:60:91:4e:9d:99:59:41:6c:57:c7:dc:12:
                    c1:64:57:75:8d:a3:64:2f:f7:e7:d4:0f:77:ea:66:
                    df:d6:f8:c9:f0:ab:ba:dd:72:6a:db:92:76:4d:be:
                    32:65:b5:8a:71:f3:b4:02:86:31:d4:ca:91:ee:70:
                    67:c7:85:a9:98:20:f0:dc:0e:02:85:f5:5f:a6:00:
                    8f:b8:4c:0f:4f:2a:c2:e8:b6:04:a4:42:10:68:d3:
                    f1:6c:a0:0c:a7:db:32:28:b5:f3:1a:1b:0a:97:8a:
                    03:a3
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                87:F6:08:88:01:EE:00:3C:4B:52:C1:A1:18:0E:B7:CE:B9:93:1A:4C
            X509v3 Authority Key Identifier:
                87:F6:08:88:01:EE:00:3C:4B:52:C1:A1:18:0E:B7:CE:B9:93:1A:4C
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        70:a4:8b:ff:2c:63:a4:48:23:5e:35:06:04:83:35:34:83:e2:
        3f:30:41:16:14:e8:5a:36:f8:71:7c:42:6a:f0:eb:fc:76:99:
        a8:b0:9f:1f:3b:00:b4:e2:d5:cf:56:02:19:11:57:5d:7e:04:
        be:7d:dc:64:8c:16:ef:0e:9a:04:c1:bd:cd:50:bc:04:44:24:
        80:c2:04:b0:fd:3f:f2:c3:92:32:4c:0e:78:36:de:09:b9:ed:
        86:da:91:d1:bd:de:16:e0:6e:65:ee:6c:bd:61:77:45:2c:e3:
        d8:80:a3:97:ba:93:8a:74:dd:6e:93:e8:5c:fb:cb:ce:79:79:
        bb:63:ac:00:cd:48:85:1f:e9:ce:d1:ad:89:4e:ba:f6:86:21:
        86:11:86:7e:75:8e:e8:81:7a:07:ea:36:c5:f9:2c:b4:3a:e6:
        a9:1c:5d:f5:7d:73:97:2d:38:8d:c9:14:91:36:5b:14:d9:61:
        9a:bf:e0:da:5c:16:08:3a:0d:21:8f:34:98:c9:8c:24:ff:d6:
        4d:b1:56:f6:df:cf:6a:99:fc:06:95:27:98:16:7d:75:85:1f:
        c0:ec:9d:3f:03:53:20:1d:66:02:aa:b7:b8:e0:aa:76:97:49:
        a8:64:f7:02:bb:0a:6e:14:8f:4e:6b:6a:d2:cc:63:87:ec:b9:
        b0:b8:73:6a

The metadata is the same, only the creation date and the actual certificate entries are different. The regular keypair works in the handshare, the tpm2 does not.

[philippun1]

The handshake actually only fails on real tpm2 hardware, it works now in a virtual machine with a simulated tpm2 module. Could there be some functionality missing on my tpm2 hardware module?

And I also debugged the specific function a little deeper and it actually goes the same way with the tpm2 certificate and the regular certificate. The XOR of EM and DB fails, in the screenshot you can see that, i.e. the first byte on each side (32 and 33) will not XOR to 0, which it expects later:

The check is done on client side, which does not use the tpm2 provider. So I assume the server side (with tpm2) sends wrong data?

Where would be the best place to look at what actually goes wrong here?

[philippun1]

I was able to test it on other TPM hardware and it works on 1 machine and does not work on 2 other machines (in addition to it working in a VM with a simulated TPM module). So in general it works, which is good.

The question now would be, how to determine which TPM module works and which do not? @gotthardp Do you maybe have an idea?

[selvanair]

Hi,

if I try to perform a TLS handshake with tpm2 provider being used on the server side on Windows, I get the following error:

SSL_connect error: error:02000088:rsa routines::salt length check failed

If it is of any help, I can also test the code on a Linux VM and see if it works there.

Any help is appreciated, even if it is only a hint on where to look at so I can debug into this myself. Thanks.

As seen in the debug window TPM is generating signature with salt length = 222 (which is the max value possible with your key and digest). But OpenSSL would want to see 32 (same as digest length) as per TLS 1.3.

This is likely because TPM on your PC is following the old standard (see this link too: https://learn.microsoft.com/en-us/answers/questions/467673/windows-10-tpm-2-0-client-authentication-in-tls-1).