paimon icon indicating copy to clipboard operation
paimon copied to clipboard

[Bug] Flink currentFetchEventTimeLag metric calculation is wrong when the datastream is completely blocked

Open xiangyuf opened this issue 1 year ago • 0 comments

Search before asking

  • [X] I searched in the issues and found nothing similar.

Paimon version

master(8ac121a)

Compute Engine

Flink

Minimal reproduce step

Check the Flink currentFetchEventTimeLag metric for a completely blocked datastream. It will not grow over time. But the currentEmitEventTimeLag will continue to grow.

What doesn't meet your expectations?

The currentFetchEventTimeLag metric should grow over time even if the datastream is completely blocked.

Anything else?

The lastSplitUpdateTime will not update for a completely blocked datastream. img_v3_02f1_22fc6ab6-2887-4dbd-94a6-fbb04603193g

In flink, the currentEmitEventTimeLag is calculated by idleTime or currentTime, so this metric can keep grow for a completely blocked datastream. img_v3_02f1_226f7c12-7838-4cac-8453-2dc9e367812g

Are you willing to submit a PR?

  • [X] I'm willing to submit a PR!

xiangyuf avatar Sep 24 '24 08:09 xiangyuf