openssl
The return value of RSA_*_{en,de}crypt() is signed
The functions RSA_(public|private)_(en|de)crypt() return a signed result, in particular -1 may be returned on error, so the caller MUST treat the value as signed.
Checklist
- [ ] documentation is added or updated
- [ ] tests are added or updated
What branches are we targeting this change for?
I hadn't looked into backports, but the code that introduced the unsigned variables to handle the signed returned values was added in 3.4 as part of https://github.com/openssl/openssl/pull/23832
So we could backport to 3.4 if that seems appropriate.
The change looks fine. Note that I am not sure how easy it is to get to this path as it happens only AFTER a successful keygen call, which normally would mean that the parameters are correct (it would have to hit a malloc failure I presume)
24 hours has passed since 'approval: done' was set, but as this PR has been updated in that time the label 'approval: ready to merge' is not being automatically set. Please review the updates and set the label manually.
