DataflowJavaSDK icon indicating copy to clipboard operation
DataflowJavaSDK copied to clipboard

Aggregators cannot be created dynamically

Open francesperry opened this issue 10 years ago • 2 comments
trafficstars

Currently, Aggregators have to be created during DoFn construction. That rules out useful cases like dynamically creating a small number of aggregators to track exceptions that get thrown during DoFn execution.

(See http://stackoverflow.com/questions/32469446/create-aggregators-dynamically-for-exceptions )

francesperry avatar Sep 09 '15 05:09 francesperry

+1

Or at least key/value based aggregator, so we can group and add according to key. So we could log each "type" of exception via key or something like that.

dovy avatar Nov 03 '15 18:11 dovy

I have latest version of Cloud SDK. I submitted a job on Google's Cloud DataFlow and I don't see it Aggregators getting created dynamically during execution. Did the fix get into Google Cloud DataFlow?

scattamanchi avatar Sep 16 '16 23:09 scattamanchi