ESP32 TFM fix for RSA key size 512 and 2048

[gojimmypi]

Description

As described in https://github.com/wolfSSL/wolfssl/issues/6205 RSA signature fails for certain key sizes on the ESP32.

This is an interim fix as described here that reverts to software calculations in the known cases that the hardware acceleration is problematic.

Specifically:

1. For key size 2048 esp_mp_mulmod is not used, instead fp_mulmod is called.

2. For key size 512 esp_mp_exptmod is not used, instead the SW version is used in fp_exptmod.

There's also a new compile-time warning these functions that need a key size but one is not defined: "WOLFSSL_RSA_KEY_SIZE not defined"

See https://github.com/wolfSSL/wolfssl/issues/6234 for a roadmap of all Espressif Improvements.

Fixes zd# n/a

Testing

How did you test?

See details in https://github.com/wolfSSL/wolfssl/issues/6205

Checklist

• [ ] added tests
• [ ] updated/added doxygen
• [ ] updated appropriate READMEs
• [ ] Updated manual and documentation

[gojimmypi]

Since this code is only for ESP32 I would remove WOLFSSL_RSA_KEY_SIZE logic and have this check always enabled and document the workaround for HW issue. The WOLFSSL_RSA_KEY_SIZE logic here is too limiting. RSA should support a range of key sizes.

I partly agree, but I don't want to disable HW acceleration for all RSA key sizes just because some are problematic. I do agree that my solution is a bit wonky, thus the "interim fix" description.

I'm going to convert this to draft and see if I can find the root problem of the HW math so that all sizes work as they should. This also means that in the meantime, any users will still encounter problems on the current release of wolfSSL, as described in https://github.com/wolfSSL/wolfssl/issues/6205

[gojimmypi]

It was definitely a worthwhile exercise to find the root cause. See https://github.com/wolfSSL/wolfssl/issues/6380

So far, I've only been testing the faster keysize = 512, so I have not determined if the TFM problems with fp_add and fp_cmp are the only ones needed for all other key sizes.

[gojimmypi]

This lingering draft PR was fixed some time ago. See also https://github.com/wolfSSL/wolfssl/issues/6205