[fix][client] Use byte schema for DLQ topic to avoid client side schema validation

[tonisojandu]

Fixes #20635

Motivation

When broker side schema validation is disabled and consumer receives an Avro message that has an incompatible schema then sending messages to DLQ fails due to a client side schema validation. This causes a fast loop that starts creating producers for DLQ topic, consumer gets stuck and explodes the number of producer metrics exposed from broker. Example implementation of this issue.

Modifications

By not inferring the DLQ producer schema from the consumer schema and just producing them as byte schema, this issue is fixed.

Verifying this change

• [ ] Make sure that the change passes the CI checks.

This change is already mostly covered by existing tests, such as org.apache.pulsar.client.impl.ConsumerImplTest.

However, to cover this specific verification edge case org.apache.pulsar.client.impl.ConsumerImpl would need to refactored to expose possibleSendToDeadLetterTopicMessages for stubbing. Alternatively, a more complicated component test would be needed that would cover the complete ConsumerImpl.messageReceived(CommandMessage cmdMessage, ByteBuf headersAndPayload, ClientCnx cnx). This would need to deal with encryption and compression. Did not find an example to base this potential test, so I am not sure what approach to take here.

Manually checked that the problem was solved in the constructed example.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• [ ] Dependencies (add or upgrade a dependency)
• [ ] The public API
• [x] The schema
• [x] The default values of configurations
• [ ] The threading model
• [ ] The binary protocol
• [ ] The REST endpoints
• [ ] The admin CLI options
• [x] The metrics
• [ ] Anything that affects deployment

Documentation

• [ ] doc
• [ ] doc-required
• [x] doc-not-needed
• [ ] doc-complete

Matching PR in forked repository

PR in forked repository: https://github.com/tonisojandu/pulsar

[github-actions[bot]]

@tonisojandu Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

[github-actions[bot]]

The pr had no activity for 30 days, mark with Stale label.