Batches fail to save for timestamps in the future

[chrisburrows]

Internally we run "metric as a service" and therefore it is possible for rogue data points to be submitted.

If align datapoint TTL is enabled, and a datapoint is sent that has a timestamp that is more than approximately 20 years in the future (and a TTL = 0), the batch will be rejected by Cassandra because the TTL is more than ~20 years.

This seems to be due to the datapoint age being calculated in BatchHandler line 104 resulting in a negative value which is then subtracted from the metric TTL resulting in the calculated TTL being very large, approximately equal to the amount of time the data point is into the future. If this is > ~20 years, Cassandra will reject the batch.

Also, the exception handler treats this in the same way as the batch being too large, so the batch size is reduced and retried. The smaller batch that contains the "bad" data point will still be rejected.

Kairos version 1.2.1.

[brianhks]

What is the behavior/error you would like to see? I understand that forever retrying bad data is, well bad.

Quick fix in your case is to create a custom plugin that listens to the data point events and rejects data with a timestamp in the future.

I just added a test to https://github.com/kairosdb/kairosdb/blob/develop/src/test/java/org/kairosdb/eventbus/FilterEventBusTest.java to show how you can filter. Basically your subscriber method needs to return the DataPointEvent you got or null if you want to reject it. Make sure to register your subscriber to have a priority less than 50 which is the default so you get it before the backend does.

[chrisburrows]

Thanks for the suggestion.

In this instance, Kairso should either discard points that have timestamps too far in the future, but how far in the future would need to be configurable to suit different use cases, and ensure the TTL values passed into Cassandra are valid so the data points are retained for as long as Cassandra will permit rather than have cassandra reject point and trigger a batch size reduction and retry.