Proper key share selection ranking

[Frauschi]

Version

master

Description

Consider an application that doesn't set a custom set of supported key exchange groups (e.g., using wolfSSL_CTX_set_groups()), for example NGINX compiled with WolfSSL. When establishing a TLS connection with a browser, multiple key shares are sent to the server (e.g. Firefox sends three in total: X25519MLKEM768, X25519 and SECP256R1).

Within the server key share selection process, for each key share, a rank value is obtained with TLSX_KeyShare_GroupRank(). In case no custom groups are set, the order within the preferredGroup array in src/tls.c is used. This results in SECP256R1 having the highest rank (as the index in the array is the rank, and a lower number is considered a higher rank). This results in the TLS server selecting the “weakest” key share (in the Firefox example, SECP256R1 is selected).

Is this the intended behavior? I think that the order in the preferredGroup array should reflect something like an actual order based on the achieved security level or something like that.