openssl
RSA decoder should check also sanity of p, q, e, d ... with repsect to n
this issue has been discovered by osss-fuzzer [1]. The test function decodes RSA key created by fuzzer and calls EVP_PKEY_pairwise_check() which proceeds to ossl_bn_miller_rabin_is_prime() check which takes too long exceeding timeout (45secs).
The idea is to fix OSSL_DECODER_from_data() code path so invalid RSA keys will be refused.
[1] https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=69134
The check should be IMO done if the key is created via a fromdata/import function as well.
Also multiprime RSA keys need to be handled.
New revision attempts to address comments raised by @t8m. In order to make evp_test to pass I had to introduce a rounding I feel uneasy about this change. The bits for key at line 610 found in test/recipes/30-test_evp_data/evppkey_rsa_common.txt were 1024. However the prime components p,q were 1025 and 1026.
there were perhaps other cases too, but I've analyzed this particular one.
this is the only friction in proposed change which I feel needs to be brought up to wider attention.
Could you add the fuzzer-generated data as a testcase somehow? IMO it should fail import with openssl rsa (or openssl pkey) with the fix. Otherwise it should pass.
Could you add the fuzzer-generated data as a testcase somehow? IMO it should fail import with openssl rsa (or openssl pkey) with the fix. Otherwise it should pass.
I've added the test to evp_extra_test.c. The key comes from fuzzer report.
