nika-qubit
nika-qubit
Thanks for filing this. The mitigation of disabling the pipeline pruning is fine. An example, we don't prune TestStream from the pipeline: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/pipeline_fragment.py#L219
It just skips the pruning for a specific transform, not the whole pruning feature. TL;DR: the pruning logic is pretty naive, it marks all transforms and input/ouptut PCollections contributing to...
> I think the "No producer" error is a bug with PCollection pruning in interactive beam: #21430 > > CC: @KevinGG Commented in https://github.com/apache/beam/issues/21430, we can disable pruning for dataframe...
> > > > I think the "No producer" error is a bug with PCollection pruning in interactive beam: #21430 > > > > CC: @KevinGG > > > >...
> > > I think the "No producer" error is a bug with PCollection pruning in interactive beam: #21430 > > > CC: @KevinGG > > > > > >...
@PhilippeMoussalli Could you please take a look at https://github.com/apache/beam/pull/23069?