[CycloneDDS/FastDDS interoprability] Public key wire representation (DH+MODP-2048-256 kagree_algo)

[lexamor]

Is there an already existing issue for this?

• [X] I have searched the existing issues

Seems #3259 can be related to this

Expected behavior

The auth/secure handshake should be passed successfully if the DH+MODP-2048-256 key agreement algo is selected.

Current behavior

In case if DH+MODP-2048-256 key agreement algorithm is selected during the secure handshake the dh1/dh2 attribute cannot be deserialized on the receiver side.

Steps to reproduce

1. Configure security (for the both fastdds&cyclone client apps).
2. Launch them

*fastdds client should be the initiator of the handshake since by default ECDH_prime256v1 can be forced by the CycloneDDS side.

Fast DDS version/commit

2.9.1

Platform/Architecture

Ubuntu Focal 20.04 amd64

Transport layer

Default configuration, UDPv4 & SHM

Additional context

According to the DDS Security spec dh1/dh2 attributes are: The CDR Big Endian Serialization of a Diffie-Hellman Public Key chosen by the Participant. This will be used for key agreement.

FastDDS implementation:

BN_bn2bin/BN_bin2bn openssl functions are used during de-/serialization routine:

// Serialization
static bool store_dh_public_key(
...
        const DH* dh = EVP_PKEY_get0_DH(dhkey);
...
        DH_get0_key(dh, &pub_key, &priv_key);
...
        int len = BN_num_bytes(pub_key);
...
        if (BN_bn2bin(pub_key, pointer) == len)`
        
// De-serialization
static const unsigned char* BN_deserialize_raw(
        BIGNUM** bn,
        const unsigned char* raw_pointer,
        size_t length,
        SecurityException& exception)
{
    // 
    BIGNUM* bnn = BN_new();

	if (BN_bin2bn(raw_pointer, static_cast<int>(length), bnn) != nullptr)
}        

CycloneDDS implementation: i2d_ASN1/d2i_ASN1 openssl functions are used during de-/serialization routine:

//Serialization
  if (!(asn1int = BN_to_ASN1_INTEGER (dh_get_public_key(dhkey), NULL)))
...
  (void) i2d_ASN1_INTEGER (asn1int, &buffer_arg);

//De-serialization
static DDS_Security_ValidationResult_t dh_oct_to_public_key_modp(EVP_PKEY **pkey, const unsigned char *keystr, uint32_t size, DDS_Security_SecurityException *ex)
{
...
  if (!(asn1int = d2i_ASN1_INTEGER(NULL, (const unsigned char **)&keystr, (long)size)))
...
  if (!(pubkey = ASN1_INTEGER_to_BN(asn1int, NULL)))
...

XML configuration file

No response

Relevant log output

CycloneDDS:
fsm: Begin handshake reply failed: Failed to convert octet sequence to ASN1 integer

Network traffic capture

No response