Tracking iterator progress

[kemsky]

Currently, example contains the following code:

var lastFromModifiedDate = response.CustomerRet.OrderBy(m => m.TimeModified).Select(m => m.TimeModified).LastOrDefault();
await dbContext.SaveIfNewerAsync(LAST_MODIFIED_CUSTOMER, lastFromModifiedDate);

I've noticed that entities received in response are not sorted and in case sync is interrupted next sync can skip some records.

Example. The first batch has lastFromModifiedDate = 01.01.2020, the second batch lastFromModifiedDate = 01.01.2019, third batch lastFromModifiedDate = 21.05.2020. So LAST_MODIFIED_CUSTOMER is now 21.05.2020, but there are few more batches pending (i.e. from 2018). If system crashes or connection drops - next sync will set request.FromModifiedDate = 21.05.2020... and miss all remaining items.

Is there any way to track sync in a safe manner?

[jsgoupil]

Just to be clear, I have never seen batches being out of order in the same CustomerQuery. Are you seeing something else here?

Regarding connection drops or system crashes. I think you might miss a couple yes, but not an entire catalog (if my first assumption is valid). I think you would have to implement flags to be fully resilient for such thing. Flags in the AuthenticateAsync and CloseConnectionAsync. Any exceptions caught with the other methods, you could flag it indicating that the last modified customer might not be correct.

[kemsky]

@jsgoupil, I've used customers as an example, actually, I need to sync Bills and I see that TimeModified is out of order (I'll double-check though).

Currently, I'm going to use two columns to store progress, the first column is updated using SaveIfNewerAsync, it stores temporary value between iterator calls, the second column is updated only when iteratorRemainingCountValue=0 and stores final value. Can I use iteratorRemainingCountValue for this purpose?

[jsgoupil]

@kemsky I think that iteratorRemainingCountValue should work. But you have to come up with a "solution" if you don't reach that iteratorRemainingCountValue to 0?

Keep us updated on the BillQuery. And if you find an elegant solution, we could integrate it in this library.

[jsgoupil]

No actions