Add RSA_PKCS1_WITH_TLS_PADDING support for legacy

[s4ex]

When SSL master key is exchanged using plain rsa encryption, the special RSA_PKCS1_WITH_TLS_PADDING mode is used to prevent Bleichenbacher's attack on PKCS #1 v1.5 RSA padding. The support for this padding type was missing from rsa_pmeth, thus causing servers which run legacy engines to fail to negotiate with the client reporing RSA_R_INVALID_PADDING_MODE error

[t8m]

OTC: This is a new feature for legacy EVP codepaths. Do we want that?

We've already previously rejected a couple of other legacy codepath enhancements.

[s4ex]

Looks like it was failing with enable-fips. Fixed that with ifndef FIPS_MODULE

[t8m]

OTC: We are OK to take this but it must have tests and documentation.

[s4ex]

OTC: We are OK to take this but it must have tests and documentation.

Fair enough. I will work on the tests and docs then.

[davidben]

When SSL master key is exchanged using plain rsa encryption, the special RSA_OAEP1_WITH_TLS_PADDING mode is used to prevent Bleichenbacher's attack on PKCS #1 v1.5 RSA padding.

I'm guessing this is a mistake in the PR description and commit message? OAEP is not the same as PKCS#1 v1.5. TLS does not use OAEP.

[mattcaswell]

Yes. Although it says that in the title/description of the PR - this is in fact about RSA_PKCS1_WITH_TLS_PADDING

[t8m]

I've edited the title and description to fix that confusion.

[s4ex]

When SSL master key is exchanged using plain rsa encryption, the special RSA_OAEP1_WITH_TLS_PADDING mode is used to prevent Bleichenbacher's attack on PKCS #1 v1.5 RSA padding.

I'm guessing this is a mistake in the PR description and commit message? OAEP is not the same as PKCS#1 v1.5. TLS does not use OAEP.

@davidben Ah.. Yeah, an embarrassing mistake from my side. Thanks for spotting this.

@t8m thanks for editing the description.

[openssl-machine]

This PR has the label 'hold: cla required' and is stale: it has not been updated in 30 days. Note that this PR may be automatically closed in the future if no CLA is provided. For CLA help see https://www.openssl.org/policies/cla.html

[openssl-machine]

This PR has the label 'hold: cla required' and is stale: it has not been updated in 61 days. Note that this PR may be automatically closed in the future if no CLA is provided. For CLA help see https://www.openssl.org/policies/cla.html

[openssl-machine]

This PR has been closed. It was waiting for a CLA for 90 days.

[t8m]

@s4ex any chance to work on the tests and docs?

[s4ex]

@t8m . Thanks for reminder. Have been busy lately. Will try to work on this in few coming weeks

[s4ex]

Working on the tests now. Maybe someone can give a recommendation - Would it be more reasonable to put it as a simple EVP RSA test, or a tls connection test ? In any case, it will require some dummy rsa engine to be used to check it works on legacy branch. Maybe there are already available which can be reused? My simple grep skills didn't bring much results...

[t8m]

@s4ex There is the dasync engine which can be used for tests.

[s4ex]

@t8m - Excellent. Thanks for the pointers.

[assurend]

In OpenSSL version 3.0.8, the TLSv1.2 handshake failed due to an error encountered while using the cipher TLS_RSA_WITH_AES_128_CBC_SHA256. This resulted in the following error message: OpenSSL: error:02000090:rsa routines::illegal or unsupported padding mode."

This issue was addressed by applying the changes mentioned in the following pull request: https://github.com/openssl/openssl/pull/22566/files

However, subsequent to the resolution of the previous error, a different error has arisen: Message: OpenSSL: error:03000093:digital envelope routines::command not supported

The following is a backtrace obtained during the second problem: Thread x "rte-worker-x" hit Breakpoint 2, pkey_rsa_ctrl (ctx=0xdcb7aef8, type=0, p1=771, p2=0x0) at ../../../../../../../src/crypto/openssl/crypto/rsa/rsa_pmeth.c:447 447 in ../../../../../../../src/crypto/openssl/crypto/rsa/rsa_pmeth.c (gdb) bt #0 pkey_rsa_ctrl (ctx=0xdcb7aef8, type=0, p1=771, p2=0x0) at ../../../../../../../src/crypto/openssl/crypto/rsa/rsa_pmeth.c:447 #1 0x00000000004ce154 in evp_pkey_ctx_ctrl_int (ctx=ctx@entry=0xdcb7aef8, keytype=keytype@entry=6, optype=optype@entry=1024, cmd=cmd@entry=0, p1=p1@entry=771, p2=p2@entry=0x0) at ../../../../../../../src/crypto/openssl/crypto/evp/pmeth_lib.c:1318 #2 0x00000000004cfc9c in EVP_PKEY_CTX_ctrl (ctx=ctx@entry=0xdcb7aef8, keytype=keytype@entry=6, optype=optype@entry=1024, cmd=0, p1=771, p2=0x0) at ../../../../../../../src/crypto/openssl/crypto/evp/pmeth_lib.c:1352 #3 0x00000000005533a3 in evp_pkey_ctx_setget_params_to_ctrl (pctx=pctx@entry=0xdcb7aef8, action_type=action_type@entry=SET, params=params@entry=0xdcb4c200) at ../../../../../../../src/crypto/openssl/crypto/evp/ctrl_params_translate.c:2749 #4 0x00000000005549cf in evp_pkey_ctx_set_params_to_ctrl (ctx=ctx@entry=0xdcb7aef8, params=params@entry=0xdcb4c200) at ../../../../../../../src/crypto/openssl/crypto/evp/ctrl_params_translate.c:2772 #5 0x00000000004cf1df in EVP_PKEY_CTX_set_params (ctx=ctx@entry=0xdcb7aef8, params=params@entry=0xdcb4c200) at ../../../../../../../src/crypto/openssl/crypto/evp/pmeth_lib.c:714 #6 0x0000000002a90c84 in tls_process_cke_rsa (s=s@entry=0xdcb4c4e8, pkt=pkt@entry=0xdcb4c330) at ../../../../../../../src/crypto/openssl/ssl/statem/statem_srvr.c:3008 #7 0x0000000002a93654 in tls_process_client_key_exchange (s=0xdcb4c4e8, pkt=0xdcb4c330) at ../../../../../../../src/crypto/openssl/ssl/statem/statem_srvr.c:3395 #8 0x0000000002a944b1 in ossl_statem_server_process_message (s=0xdcb4c4e8, pkt=) at ../../../../../../../src/crypto/openssl/ssl/statem/statem_srvr.c:1264 #9 0x0000000002a7dec8 in read_state_machine (s=s@entry=0xdcb4c4e8) at ../../../../../../../src/crypto/openssl/ssl/statem/statem.c:664 #10 0x0000000002a7e975 in state_machine (s=0xdcb4c4e8, server=server@entry=1) at ../../../../../../../src/crypto/openssl/ssl/statem/statem.c:448 #11 0x0000000002a7ec3d in ossl_statem_accept (s=) at ../../../../../../../src/crypto/openssl/ssl/statem/statem.c:276

This is a legacy mode, without any specific providers loaded explicitly: (gdb) p *ctx $1 = {operation = 1024, libctx = 0x0, propquery = 0x0, keytype = 0x0, keymgmt = 0x0, op = {keymgmt = {genctx = 0x0}, kex = {exchange = 0x0, algctx = 0x0}, sig = {signature = 0x0, algctx = 0x0}, ciph = {cipher = 0x0, algctx = 0x0}, encap = {kem = 0x0, algctx = 0x0}}, cached_parameters = {dist_id_name = 0x0, dist_id = 0x0, dist_id_len = 0, dist_id_set = 0}, app_data = 0x0, pkey_gencb = 0x0, keygen_info = 0xdcb50bdc, keygen_info_count = 2, legacy_keytype = 6, pmeth = 0x9579080 <rsa_pkey_meth>, engine = 0x0, pkey = 0xdb5d93c8, peerkey = 0x0, data = 0xdcb50bc8, flag_call_digest_custom = 0, rsa_pubexp = 0x0}

Please help to fix this issue.

[s4ex]

@assurend , thanks for reporting this. On 3.0.8 version, there is also a bug in ctrl_params_translate, because in Your backtrace, the cmd=0 when EVP_PKEY_CTX_ctrl (ctx=ctx@entry=0xdcb7aef8, keytype=keytype@entry=6, optype=optype@entry=1024, cmd=0, p1=771, p2=0x0) is called from evp_pkey_ctx_setget_params_to_ctrl. This was fixed on master in 1009940. Please try taking that commit as well.

[assurend]

Thank you very much @s4ex for your prompt response. The issue got fixed with the change you suggested.

[openssl-machine]

This PR is in a state where it requires action by @openssl/committers but the last update was 30 days ago

[openssl-machine]

This PR is in a state where it requires action by @openssl/committers but the last update was 61 days ago

[openssl-machine]

This PR is in a state where it requires action by @openssl/committers but the last update was 92 days ago

[openssl-machine]

This PR is in a state where it requires action by @openssl/committers but the last update was 123 days ago