scio icon indicating copy to clipboard operation
scio copied to clipboard

Streaming pipeline update with naming on composite transform

Open RustedBones opened this issue 1 year ago • 0 comments

When launching a replacement job for a streaming pipeline in dataflow, transformNameMapping option must be given when transformation names have changed.

Scio by default use the callsite for the transformation names, but this can be manually set calling withName before the transformation.

However, if the transformation is composite, the sub transform will keep their callsite name. If user code changes, even if they specify the name on the composite operator, they will have to provide a mapping for the sub-ones.

image

RustedBones avatar Jul 12 '22 14:07 RustedBones