elastic
OTLP/gRPC message size limit is fixed at 4MB
If the decompressed grpc message exceeds 4MB, the collector sending to apm-server logs the following error.
2024-11-25T04:08:26.561Z error internal/queue_sender.go:100 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "not retryable error: Permanent error: rpc error: code = ResourceExhausted desc = grpc: received message after decompression larger than max (6624956 vs. 4194304)", "dropped_items": 11613}
go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1
go.opentelemetry.io/collector/[email protected]/exporterhelper/internal/queue_sender.go:100
The limit comes from the default 4MB grpc server message size limit MaxRecvMsgSize in https://pkg.go.dev/google.golang.org/grpc#MaxRecvMsgSize .
Current user workarounds
- if batchprocessor is used, adjust the config values to influence the payload size
- use otlphttpexporter in place of otlpexporter in the collector sending to apm-server.
Proposed solution
apm-server should either document or adjust the limit, or make it configurable.
Do you know if there's a corresponding config in OTel collectors? Would it be sensible to mirror its name in apm-server?
Do you know if there's a corresponding config in OTel collectors?
max_recv_msg_size_mib in https://github.com/open-telemetry/opentelemetry-collector/blob/33264a5408cd918cfb45fd622b99f681c67d535c/config/configgrpc/configgrpc.go#L175
Would it be sensible to mirror its name in apm-server?
If we're adding a config, it needs to be nested under an appropriate section, as it only concerns OTLP/gRPC, not even OTLP/HTTP. We don't have any otel specific sections in apm-server.yml.
Maybe we should make it configurable, but IMO the better solution would be for clients to send smaller payloads. If the server can accept a larger payload, then it will be more susceptible to OOM.
