For how long events are stored in offline?

[khsora34]

The thing

📝 Give us the error message you receive, describe the problem and answer the questions.

We've been using the DataDog SDK on our SwiftUI app for gathering logs, traces and RUM. One of the app requirements is that it has to work even on offline environments.

Because of that, we are losing a big chunk of all the events we send to DataDog, and we are very infrequently recovering them after the user regains Internet connectivity. We configured the SDK to send as many logs as possible (trace, RUM and log trace == 100).

Our users usually use the app in offline environments a couple of days a week, so they might not reconnect until the next week. However, they always need to have Internet connection when they start the app, so we'd have time when they do have connectivity.

In our tests though, even if they do recover connectivity after a week, the events are not uploaded, so we basically are loosing all the data from the times they were offline. It seems the logs are being deleted, but we don't know what's actually happening.

What would you recommend us to do in this case to get as many logs as possible?

---

Datadog SDK version:

Which version of the Datadog SDK causes this problem? e.g. 1.2.0

2.0.0, though it's been happening since 1.17.0.

Last working Datadog SDK version:

What is the last Datadog SDK version where this problem didn't occur? e.g. 1.1.0

N/A

Dependency Manager:

Which dependency manager do you use? e.g. Cocoapods / Carthage / SPM / ...

Cocoapods

Other toolset:

Do you use additional tools with your dependency manager? e.g. CarthageCache

No

Xcode version:

e.g. Xcode 11.5 (11E608c)

Xcode 14.3.1 (14E222b)

Swift version:

e.g. 5.1

5.8

Deployment Target:

What is the Deployment Target of your app? e.g. iOS 12, iPhone + iPad

iOS 16.2, iPhone

[ncreated]

Hello @khsora34 👋. Thanks for explaining the full context. Our SDKs implement a hard limit of data obsolescence, which is designed to stay in sync with Datadog ingestion constraints, e.g. Log Collection And Integrations:

Log events can be submitted with a timestamp that is up to 18h in the past.

In practise, or SDKs delete all logs older than 18h, because they know that transporting them to Datadog server is a waste of bandwidth (these logs would get rejected anyway). We may consider extending this limit in the future to more than 18h, but it may only improve, not fix the problem you outlined.

Unfortunately, we don't offer any option today that might help you reliably send a week-old data to Datadog. In case of simple events, like Logs, you might try persisting them on your end and then passing them to our SDK next time when app is started, but that will erase their real timestamps and it may be not acceptable for you.

[khsora34]

Thanks for the answer!

We'd be interested in the way of sending the logs ourselves. Is it part of the DataDog SDK or should we send them in a different way (via an API...)? In that case, will the logs keep the same order as the one we sent them through? Maybe we can circumvent the timestamp issue by sending a timestamp ourselves when creating the log. Also, do you feel there's a way to have the best of two worlds, automatic pushing by the SDK and the rest of the logs being pushed manually?

Another way that we thought about is sending the missing logs in background. Is there a way to send the logs through the SDK in background?

Thanks again!

[ncreated]

@khsora34 if you consider changing the timestamp of logs, then I think you need to look into our POST /logs API and make the transport by yourself. Logs are always ordered by their timestamp, no matter of the order you upload them.

Also, do you feel there's a way to have the best of two worlds, automatic pushing by the SDK and the rest of the logs being pushed manually?

There's no "impossible" 😊, but it doesn't seem like simple system to build. And definitely, it's not something we can support you on - it's way too specific.

Another way that we thought about is sending the missing logs in background. Is there a way to send the logs through the SDK in background?

We don't enforce background modes. Although if your app has a reason to be awaken in the background, our SDK will leverage it by running its standard upload routines.

I must admit, it's very uncommon set of constraints that you're having, but I totally understand it 👍. I hope my answers can give you something.

[khsora34]

Yeah, thank you for your insights, they are really helpful.

Just wanted to ask you more about the POST /logs approach. It seems it still won't accept any logs older than 18 hours.

Log events can be submitted with a timestamp that is up to 18 hours in the past.

Do you know if that holds true? Just verifying, in order to check how to send the logs that were missed.

[plousada]

Hi @khsora34 , yes the same requirements will still apply.

If you wanted to send logs that were over 18h you would have to persist them in your app, and then once the client regained connectively re read them, alter their timestamp to be within the 18h limit, and send the via POST API.

In this case you will lose the real/original moment in time the log occurred. But, as an alternative you could potentially add them as attributes to the log.