opentelemetry-erlang-contrib icon indicating copy to clipboard operation
opentelemetry-erlang-contrib copied to clipboard

No way to set max compression size limit

Open sashaafm opened this issue 2 years ago • 3 comments

Describe the bug Maybe not really a bug but more a lack of configuration or adherence to the spec.

We're seeing this kind of error:

OTLP grpc export failed with GRPC status 8 : grpc: received message after decompression larger than max (13104533 vs. 4194304)

Which looks like the message is too large and hits the 4194304 size limit. However, some people like in this issue here seem to have configurable ways of increasing this limit.

Possibly a workaround is to play with these env vars decreasing their scheduled delay and max queue size to deliver messages quicker before they reach this size limit?

Expected behavior A way to avoid this error, either by a new configuration or some documentation explaining how to tackle it.

Additional context

sashaafm avatar Mar 14 '23 14:03 sashaafm

@sashaafm you are seeing this error in tempo?

tsloughter avatar Mar 14 '23 17:03 tsloughter

No, this happens in the Elixir console. We’re using Datadog as a metrics aggregator/visualizer. However, this happens in the instance where the app is running (alongside the Datadog Agent).

sashaafm avatar Mar 14 '23 17:03 sashaafm

Adding here what I said in Slack so it isn't lost in time :)

We don't yet support OTEL_BSP_MAX_EXPORT_BATCH_SIZE but do have OTEL_BSP_SCHEDULE_DELAY_MILLIS (bsp_scheduled_delay_ms in the application configuration) which defaults to 5 seconds.

tsloughter avatar Mar 15 '23 19:03 tsloughter