ECDSA verify fails on windows when R,S values both have leading zeroes.

[shane-kearns]

Initially observed that some ECDSA certificates are considered to have invalid signatures by certvalidator but which are valid according to windows certutil.exe and openssl.

The root cause is the marshalling of R, S values for calling BCryptVerifySignature here: https://github.com/wbond/oscrypto/blob/1547f535001ba568b239b8797465536759c742a3/oscrypto/_win/asymmetric.py#L2629-L2649 Because oscrypto passes a 62 byte signature where a 64 byte signature is expected (in the case of NIST p256 keys), windows returns STATUS_INVALID_PARAMETER, which is treated as an invalid signature at line 2647.

The DSASignature.to_p1363 function does not know the expected key size: https://github.com/wbond/asn1crypto/blob/b763a757bb2bef2ab63620611ddd8006d5e9e4a2/asn1crypto/algos.py#L720-L736 It can't, since the (EC)DSA signature ASN1 is just a sequence of two Integers. The context of what the signature algorithm and key size are is held at a higher level.

The required length can be determined from the public key used for verification. e.g. certificate_or_public_key.byte_size is 32 for an ECDSA p256 key (since it's encoded as a compressed point), which is also the size of the private key and thereforre the required size for each of the R and S values.

Following script usually reproduces the problem within the 100k iterations

from asn1crypto.algos import DSASignature
from oscrypto import backend, platform
from oscrypto.asymmetric import ecdsa_sign, ecdsa_verify, generate_pair
from oscrypto.errors import SignatureError
import secrets

def check():
    public, private = generate_pair("ec", curve="secp256r1")
    data = secrets.token_bytes(128)
    signature = ecdsa_sign(private, data, "sha256")
    try:
        ecdsa_verify(public, signature, data, "sha256")
    except SignatureError:
        print("\n\nBad signature!")
        print("public key:", public.asn1.dump().hex())
        print("private key:", private.asn1.dump().hex())
        print("data:", data.hex())
        print("signature:", signature.hex())
        p1363 = DSASignature.load(signature).to_p1363()
        print(f"P1363 signature: {p1363.hex()} ({len(p1363)} bytes)")
        raise

print("testing on", platform.platform(), "with backend", backend())
for i in range(100000):
    if(i % 7800 == 0):
        print()
    if(i % 100 == 0):
        print('.', end="", flush=True)
    check()