KAFKA-14149: Fix DynamicBrokerReconfigurationTest in 3.2

[mimaison]

@mumrah @cmccabe It looks like a way to fix DynamicBrokerReconfigurationTest in 3.2 is to backport https://github.com/apache/kafka/commit/9c3f605fc78f297ecf5accdcdec18471c19cf7d6. I think it makes sense to to also backport https://github.com/apache/kafka/commit/78038bca6688ce01f7df238d53142f5de3455863, otherwise testKeyStoreAlter() is really flaky.

Before I push to 3.2, I want you to take a look and confirm if this makes sense.

Please do not merge this PR, instead I'll rebase directly the commits as we don't want to squash them.