scio icon indicating copy to clipboard operation
scio copied to clipboard

Published 20 hours ago • verified publisher icon spotify

Using sparkey largeHashIntersectByKey blocks pipeline on Dataflow v1

Open RustedBones opened this issue 1 year ago • 0 comments

When using largeHashIntersectByKey, we've experienced pipelines hanging when running with dataflow default runnuer.

It seems that enabling the runner v2 unblocks the situation, but we should check why sparkey blocks worker progression.

RustedBones avatar Dec 12 '23 10:12 RustedBones