feathub icon indicating copy to clipboard operation
feathub copied to clipboard

Optimize sliding window late data insertion with skip list

Open yunfengzhou-hub opened this issue 11 months ago • 0 comments

SlidingWindowKeyedProcessFunction.java:

                if (timestamps.isEmpty() || timestamps.get(timestamps.size() - 1) < timestamp) {
                    timestampList.add(timestamp);
                } else {
                    // TODO: Use skip list to optimize the performance of inserting late data.
                    for (int insertionIdx = 0; insertionIdx < timestamps.size(); insertionIdx++) {
                        if (timestamps.get(insertionIdx) < timestamp) {
                            continue;
                        }
                        timestamps.add(insertionIdx, timestamp);
                        break;
                    }
                    timestampList.update(timestamps);
                }

yunfengzhou-hub avatar Sep 18 '23 02:09 yunfengzhou-hub