Retry 413 / payload too large errors in OTLP batch exporter

[adriangb]

Is your feature request related to a problem?

Many backends have size limits. If you try to send too large of a payload it may get dropped on the floor. This is somewhat mitigated by setting a batch size limit (in terms of number of spans) but that (1) is not at the byte size level and (2) you need to know ahead of time what your backend's limit is.

Describe the solution you'd like

I would the SDK to retry on this failure code by reducing the payload size (e.g. split it into two exports) until it's accepted or a single span is rejected.

Ideally we'd have a way for the backend to communicate back it's limit and the SDK could even store that and avoid hitting the error in the future, but I think that would require agreement at the OTEL spec level for an expected response payload body that includes the maximum allowed size. Implementing a retry in the Python SDK can be a good step in that direction.

Describe alternatives you've considered

No response

Additional Context

No response

Would you like to implement a fix?

Yes