Tomer Kaftan

Ecosystem review notes: We think addons would be a better home for this than tf core. It also does not conflict with plans for core tf.

Notes from ecosystem review: Rather than an optimizer that wraps another optimizer, we think this might actually make sense as an object to use as a gradient_transformer in optimizers (a...

Took this off of Release 0.1 after some discussion with @etrain, because there are some licensing issues we need to figure out due to how jAudio is being used.

What percent of these are only being used in the "fitting" part of a pipeline and not the "prediction" part?

And are these all RDD to RDD?

So after taking a closer look, it seems to me like the cases we're using FunctionNode right now fall under either: - Some form of 'aggregation' (representable as any RDD...

I think it should be able to work the same way the GatherTransformer works: a TransformerNode that takes multiple RDDs together as input.

I think there are a few common patterns we've seen so far: - directory structure (each item in a line so we can load it w/ sc.textfile, each item a...

It's probably best to either encourage storing data in a certain way and pick a single faster pattern to deal with, or to somehow allow mixing and matching among these....

@etrain @shivaram So I think I’ve finally come up with a valuable insight for keeping the pipeline interface linearized even when doing complex DAG operations. (Important because the linearized interface...