scylla-cdc-java icon indicating copy to clipboard operation
scylla-cdc-java copied to clipboard
verified publisher icon scylladb

Use long windows when catching up

Open avelanarius opened this issue 4 years ago • 2 comments

Changes the strategy the CDC library uses to generate windows to read the CDC log. Previously, when a library was started, it would read queryTimeWindowSize large windows starting from |now - ttl|. The issue with that approach is that it requires a large number of windows to catch up, for example: 24 hours / 30 seconds * 256 * 8 = 5898240 for a 8-shard Scylla server.

A new approach just creates a single large window from |now - ttl| to |now - confidence window|.

avelanarius avatar Feb 25 '22 05:02 avelanarius

v2: fixed flaky unit test. (off-by-one error)

avelanarius avatar Mar 10 '22 12:03 avelanarius

Any news here? I'm pretty interested in this PR

racevedoo avatar May 16 '22 13:05 racevedoo