"RSA-keyGen-FIPS186-5" generates incorrect values for p, q since it ignores pMod8 and qMod8 settings.

[slontis]

environment DEMO

testSessionId ?

vsId 3348752

Algorithm registration

{
    "algorithm": "RSA",
    "revision": "FIPS186-5",
    "mode": "keyGen",
    "prereqVals": [
        {
            "algorithm": "SHA",
            "valValue": "same"
        },
        {
            "algorithm": "DRBG",
            "valValue": "same"
        }
    ],
    "infoGeneratedByServer": true,
    "pubExpMode": "fixed",
    "fixedPubExp": "010001",
    "keyFormat": "standard",
    "capabilities": [
        {
            "randPQ": "probableWithProbableAux",
            "properties": [
                {
                    "modulo": 2048,
                    "hashAlg": [],
                    "primeTest": [
                        "2pow100"
                    ],
                    "pMod8": 1,
                    "qMod8": 3
                },
                ....
            ]
        }
    ]
}

Endpoint in which the error is experienced ?

Expected behavior

1. It is expected that the output p & q should be different if "pMod8" and/or "qMod8" are set. Currently the result for the output p & q corresponds to "pMod8" and "qMod8" being 0
2. Also "probableWithProbableAux" should output fields for p1, p2, q1, q2

Additional context

Initially I used the ACVP-SERVER/gen-val/json-files/RSA-KeyGen-FIPS186-5/internalProjection.json which uses gen-val/json-files/RSA-KeyGen-FIPS186-5/registration.json

i.e.
"randPQ": "probableWithProbableAux",
 "pMod8": 3,
 "qMod8": 5

And found that the output it produced corresponded to 0,0 instead of 3,5. I tested it here https://github.com/openssl/openssl/pull/28349.

Data was also generated using lib_acvp (abkarcher kindly generated data for me) and this also generated incorrect data.

[celic]

Hi @slontis, you're right, the values are not being respected. When the RSA KeyGen FIPS 186-5 code was set up, nothing linked the parameter capability to the crypto, so the value was essentially ignored.

Additionally that example registration is unnecessarily confusing because pMod8 and qMod8 do not apply to provable prime generation methods. I have cleaned that up locally. I am testing generation of a "fixed" vector set using that example registration. Once I can confirm everything looks good, I'll drop the files here.

[celic]

Here is an updated internalProjection.json for the same registration. Thank you for the bug report.

internalProjection.json

[celic]

On the second item, for probableWithProbableAux, you're asking for p1, p2, q1, q2. These are not included in any of our files by default. The system will provide Xp1, Xp2, Xq1, Xq2 because these are inputs to the function from a DRBG, but the p1, p2, q1, q2 are intermediate values and not needed for the validation. If you obtain the correct p, q the rest will also be correct. Did you need them for testing purposes?

[slontis]

Did you need them for testing purposes?

No. I assume historically they must of been part of the expected values from a old CAVP test (otherwise I would not have added them to my source).

[slontis]

I am testing generation of a "fixed" vector set using that example registration. Once I can confirm everything looks good, I'll drop the files here.

Thanks for doing that. I can confirm that I got the same p, q values for the "probableWithProbableAux" case that you generated.

[celic]

Do you want a set of tests with the extra intermediate values? I'd generate them manually. If they aren't for a validation, I don't think it's worth the trouble to add them in every vector set.

[slontis]

Do you want a set of tests with the extra intermediate values? I'd generate them manually. If they aren't for a validation, I don't think it's worth the trouble to add them in every vector set.

No its not necessary - I was just pointing out that the output has changed from what it used to be with the older CAVP data.

[livebe01]

The fix for this is on Demo in release v1.1.0.41.

[livebe01]

As of 11/12/25, the fix for this is on Prod in release v1.1.0.41.