Carson Ip

Sorry for the late reply. I believe we should still quickly audit the code to check for similar issues across the codebase. These will be code that manipulate global variables...

> Do you know if there's a corresponding config in OTel collectors? `max_recv_msg_size_mib` in https://github.com/open-telemetry/opentelemetry-collector/blob/33264a5408cd918cfb45fd622b99f681c67d535c/config/configgrpc/configgrpc.go#L175 > Would it be sensible to mirror its name in apm-server? If we're adding a...

Great question. I've update the description to mention user ingest pipelines.

@up2neck thanks for the bug report. I managed to reproduce it. apm-server request log ```json {"log.level":"error","@timestamp":"2024-12-07T00:18:06.391Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":59},"message":"request timed out","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"curl/8.5.0","source.address":"127.0.0.1","trace.id":"7d605e1eeeed169c7580ea71d20cfa3c","transaction.id":"7d605e1eeeed169c","http.request.id":"7d605e1eeeed169c","event.duration":266192314,"http.request.body.bytes":303694,"http.response.status_code":503,"error.message":"request timed out","ecs.version":"1.6.0"} ``` apm-server self-instrumentation trace document ```json { "_index": ".ds-traces-apm-default-2024.12.06-000001", "_id":...

I've updated the above comment to avoid some confusion. ~~TLDR of the issue is that because of ES backpressure, APM agent request times out and terminates the connection. apm-server TimeoutMiddleware...

@inge4pres raised some questions about my explanation and we managed to get to the bottom of it. I've crossed out some incorrect parts in my previous comment. I had a...

It was intentional to have `sampling.tail.discard_on_write_failure` undocumented, as it was supposed to be an escape hatch for users who are facing TBS storage limit issues while we look for a...

go-docappender itself exposes metrics as OTel metrics, as opposed to elastic agent monitoring where EA polls metrics from APM server. The go-docappender metrics should be available if `instrumentation.enabled` is true...

I looked into this again. apm-server does get stats from the appender and publish libbeat monitoring metrics, other than the fact that go-docappender publishes OTel metrics itself (what I mentioned...

Might be related to https://github.com/elastic/apm-server/issues/8383