flink-remote-shuffle icon indicating copy to clipboard operation
flink-remote-shuffle copied to clipboard

Published 20 hours ago • verified publisher icon flink-extended

Handle network issues gracefully

Open wsry opened this issue 3 years ago • 0 comments

Motivation

Currently, network issues like unstable network may cause task failover which may further lead to reproducing of data. In fact, we can improve the behavior by reconnecting and retransmitting to the remote ShuffleWorker.

Changes

For network issues, the client and server should not fail immediately, instead, if the client can reconnect to the server in some timeout, no failover or data reproducing should be triggered. At the same time, we may need a switch which can disable this feature.

Test

  • Unit test.
  • Test manually on a cluster.

wsry avatar Dec 06 '21 10:12 wsry