DTLS certificate not hashed with same algorithm as fingerprint

[SamVanBelle]

Describe the bug

Secure audio between pjsip and TE-System's anynode SBC currently fails. The SDP-fingerprint attribute is hashed with SHA-256, but the pjmedia.pjsip.org DTLS certificate is always hashed with sha1.

Reply from TE-Systems support: RFC 4572 requires (see page 8) that a certificate fingerprint [for the fingerprint attribute in SDP] MUST be computed using the same one-way hash function as is used in the certificate's signature algorithm. RFC 5763 (also page 8) imports the same restriction into the DTLS/SRTP realm.

Please update the ssl_generate_cert method in transport_srtp_dtls.c

Steps to reproduce

Make an SAVP call via anynode SBC to a pjsip app.

PJSIP version

2.13

Context

Tested on Windows.

Log, call stack, etc

See screenshots in context

[trengginas]

Could you elaborate on the fail message? Have you tried changing the code in https://github.com/pjsip/pjproject/blob/e12a8d17a02b14ec9d77de6221324d1d0d03fe49/pjmedia/src/pjmedia/transport_srtp_dtls.c#L467 to use EVP_sha256() instead of EVP_sha1()? Note that although the certificate signing is using sha1, the sha256 digest is still retrievable (e.g.: used on SDP) at least using OpenSSL.

[SamVanBelle]

Please find attached pcap between pjsip based IVR (10.0.0.190) and anynode SBC (10.0.0.237). Due to the issue with the certificate, anynode does not send audio to the IVR, and sends a DTLS alert after 10 seconds. Also attached is the extracted pjsip certificate. test6 IVR with secure audio fails.zip

I'm wondering if the certificate signing can stay hardcoded. Shouldn't it be made dependent on how the fingerprint is hashed?

[trengginas]

Currently, there's no setting to specify the hash method. The issue is that the certificate is signed using sha1 and the SDP is using sha256. The options:

1. use sha256 to generate the certificate, or
2. use sha1 on the SDP

Could you try the patch and see if the issue persists. patch_dtls_sha1.patch patch_dtls_sha256.patch

[sauwming]

I'm wondering if the certificate signing can stay hardcoded. Shouldn't it be made dependent on how the fingerprint is hashed?

It should be okay to hardcode the signature algorithm since it's to sign our local certificate. And it should be the other way around, later the fingerprint computation needs to use the same hash algorithm as the one used for signing.

I have created PR #3876 for this.