Application ends before sending queued events

[marcoregueira]

I'm using this agent in monitoring a bunch of scheduled "one shot" services whose process ends when there is not more work to do until launched again. I'm losing a lot of interesting events because they're lost when the program ends.

I need to wait until LocalPayloadSenderV2 has finished sending all events in the queue. But many times the process finishes too soon for any events to be sent.

In my case, I wrap the entire process in one transaction, this transaction completes at the end of the process and is never sent. This global transaction is really interesting to me to for checking performance.

I wouldn't mind checking the queue for a while to see if there are any events pending and they're being sent, but I can't do that because the queue is internal and I can't get any reference.

It would be great having some method to force flushing events and wait until they're sent. Knowing how many of them are in the queue would be even better.

[marcoregueira]

I've set the label as bug instead of "feature request" because in many cases I'm no receiving any transaction records, and, to my understanding, that's something that shouldn't happen.

[gregkalapos]

Thanks for opening this.

This is somewhat a duplicate of https://github.com/elastic/apm-agent-dotnet/issues/288 with some new info here.

I wouldn't mind checking the queue for a while to see if there are any events pending and they're being sent, but I can't do that because the queue is internal and I can't get any reference.

That's actually an interesting and actionable point. I think until now we were think about a solution that holds the process until every transaction was processed (either sent, or sending was tried and failed), which is not trivial to implement.

But offering an API and let user code hold the process until the queue is not empty would be a fairly easy thing to implement.

[marcoregueira]

Thanks for your answer.

I agree that it would be hard to keep the application working until the queue is cleared...

Polling the queue for status is a small price to pay. I plan to send a certain number of events, so at a certain interval I would check the queue to see if the length is reducing in a loop. If it is not, I would exit anyway.

I think that's the minimum use case.

Something crazy came up to mind while writing this.... An event when sending records to APM could be great too, to see if records are going out... In the meanwhile, in order to simulate that event, I'm going to provide a custom replacement for your ConsoleLogger and check for any events like "Sent items to server%". If there are any, I will wait another interval. If there are not, I will quit waiting.... I'm quite certain that will work for now. Unfortunately only the addToQueue operation saves a structured log with the current queue status, sending doesn't do... but that will do the trick by now...

[mathis-m]

Have written PoC to show how to integrate into .Net Core Console Application. Solved extending IServiceCollection to provide Elastic.APM Dependencies using Microsoft.Extensions.DependencyInjection. Solved construction via IConfiguration. Solved waiting for Agent to flush the event queue.

[devree-bvb]

Hi, is there any progress on this issue? The solution that you can have an optional API call that blocks until all logging has been processed would be fine. Currently, when we try to use the apm agent in a console application, we often lose data and the logging in Elastic is incomplete, which makes it unusable.

Or is the purpose of the agent only asp.net applications? That could be a design decision of course of the team, then we would have to look at other monitoring tools / libraries for these console types of applications.