return value issues

[JohnLegrandRichards]

trafficstars

PKCS11 RSA signature is returning a bit string or something. Is there a way to specify the format of the signature that gets returned?

[larskanis]

A RSA signature is just a big number represented as a byte string. It should have the size of the RSA modulus, which is 256 bytes for a 2048 bit RSA key for instance.

There is CKM_RSA_PKCS which is a raw RSA signature without any digest computation. In contrast CKM_SHA256_RSA_PKCS does some precalculation like so before building the RSA signature:

          # Build the digest info TLV as PKCS#1
          ary1 = [
            OpenSSL::ASN1::ObjectId.new('SHA256'),
            OpenSSL::ASN1::Null.new(nil),
          ]
          ary2 = [
            OpenSSL::ASN1::Sequence.new(ary1),
            OpenSSL::ASN1::OctetString.new(OpenSSL::Digest::SHA256.new(document_to_be_signed).digest),
          ]
          raw_value_to_be_signed = OpenSSL::ASN1::Sequence.new(ary2).to_der

Similarly for CKM_SHA256_RSA_PKCS_PSS, which has a bit more complicated precalculation and padding.

Simple raw RSA signature is also in our tests: https://github.com/larskanis/pkcs11/blob/eeda7cbe721af0c4a0c6535a1fe377884c4d6ef0/test/test_pkcs11_crypt.rb#L72-L89