DataflowJavaSDK
DataflowJavaSDK copied to clipboard
Aggregators cannot be created dynamically
Currently, Aggregators have to be created during DoFn construction. That rules out useful cases like dynamically creating a small number of aggregators to track exceptions that get thrown during DoFn execution.
(See http://stackoverflow.com/questions/32469446/create-aggregators-dynamically-for-exceptions )
+1
Or at least key/value based aggregator, so we can group and add according to key. So we could log each "type" of exception via key or something like that.
I have latest version of Cloud SDK. I submitted a job on Google's Cloud DataFlow and I don't see it Aggregators getting created dynamically during execution. Did the fix get into Google Cloud DataFlow?