resolved: allow configuring DNSSEC signature algorithms and digests

[msekletar]

Filing as a draft to get initial idea from folks about the general approach.

Btw, this was lightly test on my machine and with the patch resolved starts and it seems to work as expected but I haven't spent too much time trying to crash it.

[poettering]

why? i mean, the DNS servers publish what the DNS servers publish, we have to work with that

[msekletar]

System-wide policy for crypto libraries might be such that some algorithm is disabled and library returns error (e.g. RSASHA1 by default on RHEL-9) and then we would also return resolution error when DNSSEC=yes. But with this option (e.g. DNSSECAlgorithms=~RSASHA1) you can soften this and just return "authenticated: no" validation result instead of failing resolving. DNSSEC validators in bind and knot implement similar behavior.

[msekletar]

Updated. @poettering @keszybz PTAL.

[yuwata]

1191s ../src/resolve/resolved-dns-query.c: In function ‘dns_query_accept’:
1191s ../src/resolve/resolved-dns-query.c:944:27: error: ‘r’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
1191s   944 |         q->answer_errno = -r;
1191s       |                           ^~
1191s cc1: all warnings being treated as errors

This is already fixed by 5169f8cfd543cbce498871527ea8aa454ff2c252. Please rebase.

[keszybz]

We discussed this again, and the conclusion was that it does not play nicely with the global openssl policy. In resolved, we could limit what algorithms are supported, but we cannot bring back the algorithms that were disabled by policy. So anyway, the user will have to configure the policy, and it's not useful to provide a second layer of configuration.

If openssl would allow the policy to be configured by the application, doing this here would make a lot more sense.