[PIP-338] Add default lookup listener and fix inconsistency with listener's usage between different protocols

[Meet0861]

Motivation

A pip proposal to add a new default lookup listener config and fix bug with listener usage's for http protocol.

Issue in existing Code flow Currently, the listener mentioned as internalListenerName in broker config helps in deciding the listener from the list of advertisedListeners to specify the service URL, is being used for broker-to-broker communication but that is exposed to the client in case of lookup results or redirects as well.

Even if the client sends a listenerName (PIP-95) corresponding to the http protocol broker’s addresses, it is only used in the pulsar and pulsar+ssl protocols and is not consistent for the other protocols.

Need to have lookupListener It might not be feasible to have the listerner config at the client side in every tech stack or connector. Also, Currently, there is no option to pass the listener while using the admin APIs. As admin APIs can be called from an external network as well, the use of the internal listener’s broker service URL can lead all admin operations to get affected.

Modifications

1. Fix Code Flow - Inconsistency with listener usage with protocols other than pulsar and pulsar+ssl
2. Add a new config to be considered in case of missing listener in client request and return the broker service URL corresponding to the lookupListener

Verifying this change

• [ ] Make sure that the change passes the CI checks.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• [ ] Dependencies (add or upgrade a dependency)
• [x] The public API
• [ ] The schema
• [ ] The default values of configurations
• [ ] The threading model
• [ ] The binary protocol
• [ ] The REST endpoints
• [ ] The admin CLI options
• [ ] The metrics
• [ ] Anything that affects deployment

Documentation

• [ ] doc
• [x] doc-required
• [ ] doc-not-needed
• [ ] doc-complete

[Meet0861]

@codelipenghui @eolivelli @EronWright @wangjialing218 Can you please help here with a review?