apm-agent-dotnet
apm-agent-dotnet copied to clipboard
elastic
Act on server responses and implement back-off
Currently the agent tries to send each event once, and if it fails, it just logs.
We should implement a retry logic and listen to responses from the server and act on those responses.
From: https://github.com/elastic/apm-agent-dotnet/pull/136#discussion_r264652202
Related: https://github.com/elastic/apm-agent-java/pull/181 https://github.com/elastic/apm-agent-java/pull/210
I think we decided that there will be no retries in sending messages, if the message send fails it will get lost, the backoff is for the next in the queue.
Please check with other agents to make sure we have a common approach.
See https://github.com/elastic/apm/blob/master/docs/agent-development.md#transport-errors
When agent receives unsuccessful HTTP status code from APM Server we should log not just the body of the response but HTTP status code itself as well. For example in #433 - we logged
{PayloadSenderV2} Failed sending event. queue is full
and logging HTTP status code as well would have made it easier to find what condition among the ones listed at https://www.elastic.co/guide/en/apm/server/7.3/common-problems.html triggered the response.
https://github.com/elastic/apm/blob/master/docs/agent-development.md#batchingstreaming-data is 404, is there any other update on this? FlushInterval is not working for me. It flushes randomly and really slow.
That part of the doc got moved here.
No update on implementing back-off. We haven't done that.
What's exactly your issue @wast?
Well I got here while looking for FlushInterval and performance issues. I'm having a problem with rare Flushes.
My single transaction is quite big, it has 10 spans and I'm running it at localhost currently. When I turn on small load test like 100 RPM, APM server doesn't get data from my app.
Look at "newEventQueueCounts" and timing of "{PayloadSenderV2} Sent items to server".
Current config: "TransactionSampleRate": 0.2, "StackTraceLimit": 10, "FlushInterval": "5s", //doesn't work "MaxQueueEventCount": 5000, "SpanFramesMinDuration": "100ms", //this doesn't seem work, all spans including smaller than 100ms are recorded "CaptureHeaders": true, "CentralConfig": false, "CloudProvider": "none"
Let me move this to a separate issue - it seems to be unrelated to this one. I opened https://github.com/elastic/apm-agent-dotnet/issues/1210 and I'll try to help there.