euphoria icon indicating copy to clipboard operation
euphoria copied to clipboard

euphoria-flink: do not send periodic watermarks when not necessary

Open xitep opened this issue 9 years ago • 1 comments

By default, the streaming flink executor periodically forwards ingestion time watermarks from the sources. in case no windowing is part of a flow, this generates some unnecessary traffic; we can see this traffic in flink's task tracker ui.

Before translating a euphoria flow into a flink DAG, we can analyze the flow, and determine whether sending periodic watermarks starting here (i.e. InputTranslator) is necessary. If not setting the flows streamTimeCharacteristic to "processing time" might do the trick.

xitep avatar Feb 24 '17 12:02 xitep

Since pull request #59 Flink executor runs using ingestion time characteristics. Timestamp is assigned to element implicitly by Flink in SourceFunction, not in InputTranslator anymore.

vanekjar avatar Mar 27 '17 08:03 vanekjar