keystone icon indicating copy to clipboard operation
keystone copied to clipboard

Create a union operator

Open etrain opened this issue 8 years ago • 1 comments

In addition to the gather operator which concatenates horizontally, it would be nice to have a union operator which concatenates veritcally.

This came up in a discussion with @shivaram today around data augmentation.

etrain avatar Mar 15 '16 04:03 etrain

Just to add the idea here is that similar to Pipeline.gather we can express data augmentation as something like

val data = CifarLoader(trainLocation)
val featurePipeline = Pipeline.concat {
  RandomPatch() andThen
  RandomFlip() andThen
  Convolver andThen
  Pooler
  ...
}
// Here the RDD that is passed in to the next stage of the pipeline has the union
val solvePipeline = featurePipeline andThen BlockWeightedSolver()

shivaram avatar Mar 15 '16 04:03 shivaram