Feature: Give Producer More Control over Handling Schema Errors

[rnowling-memphis]

Current behavior

When using the Python SDK, if a message fails schema validation, an exception is raised.

I see a couple of issues:

1. This is a bit confusing to the developer. Usually, an exception would indicate that the operation failed (e.g., the message was never sent).
2. This API doesn't provide the developer with an opportunity to decide how to respond. For example, maybe the developer wants to decides not to send the message. Or maybe the developer wants to send the message anyway to respect the configured policy on the broker.

Suggested solution

I would start by defining policies on both the broker and producer sides. For an example, the broker can:

1. Not enforce the schema (accept all messages).
2. Do not accept messages that failed the schema validation.
3. Redirect messages to the dead letter queue.

If a schema violation is raised, the producer could throw an exception and allow the programmer to decide whether to send the message anyway. What ultimately happens will be a result of the combination of the broker policy and producer actions. If the producer decides to send the message anyway, then:

1. If the broker doesn't enforce the schema, the message will be sent to the station.
2. If the broker ignore messages that failed the schema validation, then the message is not delivered. (The producer may raise an exception indicating that the message cannot be sent.)
3. If the broker is configured to redirect messages to the dead letter queue, then the message is delivered to the dead-letter queue.

By implementing the API this way, the producer will have greater control over how to handle errors. For example, a producer may not care at all -- it lets the broker decide the policy by attempting to send all messages with schema errors and silently ignores an exception if the messages can't be sent. If sending a batch of messages, the producer may return the messages that had schema violations but successfully send those that don't. Or, the producer may propagate the error upstream to whatever code is calling it.

Additional context

No response

Code of Conduct

• [x] I agree to follow this project's Code of Conduct

Contributing Docs

• [x] I agree to follow this project's Contribution Docs