Memory Limiter errors are not retried by Receiver

[ledor473]

Describe the bug I'm building a Receiver and wanted to make it safe to use alongside the Memory Limiter but it appears that the way the Memory Limiter works versus how it's documented differs.

Steps to reproduce N/A

What did you expect to see? Based on the documentation, I would expect that if the Memory Limiter kicks in, the Receiver would retry to send it:

In memory limited mode the error returned by ConsumeLogs/Trace/Metrics function is a non-permanent error. When receivers see this error they are expected to retry sending the same data. […] Warning: if the component preceding the memory limiter in the pipeline does not correctly retry and send the data again after ConsumeLogs/Trace/Metrics functions return then that data will be permanently lost. We consider such components incorrectly implemented.

Also, I would think we need to export or have a way to match the error returned by the Memory Limiter so the Receiver can tell if the error is retriable or not.

What did you see instead?

• In the OTLP Receiver if an error is returned by nextConsumer.ConsumeLogs it appears to be directly sent back to the client rather than retried.
• In the FanOut Consumer, it appears to behave similarly and not retry error from Memory Limiter

What version did you use? I'm looking at the source code of v0.78.2

What config did you use? N/A

Environment N/A

Additional context I started a thread in CNCF Slack but figured it would be best to use Github Issues: https://cloud-native.slack.com/archives/C01N6P7KR6W/p1685721182060179