microsoft
Sender should read backend response to properly separate accepted/rejected items before sending to disk retry (Backend Response 206)
Before supporting this, the logic to read telemetry from disk needs to have a check for payload age
@OsvaldoRosado When you say "check for payload age", I'm guessing this is due to app insights not ingesting telemetry over 48 hours old? May I ask what should be the expected behaviour here for telemetry > 48hours?
We are currently blocked on using app insights for a project because of this limitation, as the servers which the apps are installed on are expected to go offline for over 48 hours on a frequent basis. The current behaviour of the app insights SDK appear to ignore stale logs, which remain in the cache folder indefinitely. Would it be better to save these stale logs into a different folder instead so that they can be still be managed? (e.g. using the OMS agent or data collector API to send it to a Log analytics workspace)
Our team is looking at making a PR on this... but would be great to get your insight first on appropriate handling of this scenario.
Sorry on second look at the source code, seems like the stale logs would get deleted. But would still like to ask the same question of whether storing stale logs in a separate folder instead of deletion would be an option.
The expected behavior is indeed to drop the file. As for redirecting stale items - I don't think the SDK currently can distinguish the backend failing a payload due to age vs some other reason (eg. invalid data) and will treat them equally for deletion. Changing this might require deserializing the payload loaded from disk and inspecting the timestamps within (basically the same problems limiting support for 206).
A workaround here might be to supply your own implementation/subclass of Sender to your TelemetryClient instance with changed failure behavior. TelemetryClient should allow swapping this via client.channel.sender = new MySender(). I expect this would let you fully customize any failure behavior needed.
@markwolff Any other ideas?
As Osvaldo mentioned, the best way seems to bake your own class which can extend the Sender class. You can implement custom behavior based on payload age (on any retriable response code) to move stale data to wherever you need it to be.
https://github.com/microsoft/ApplicationInsights-node.js/blob/369b91d7c4f25a17a55671c60394edd9d9baf325/Library/Sender.ts#L147-L148
@OsvaldoRosado @markwolff Ok, got it. Thank you very much for the recommendation both!