keystone
keystone copied to clipboard
Create a union operator
In addition to the gather operator which concatenates horizontally, it would be nice to have a union operator which concatenates veritcally.
This came up in a discussion with @shivaram today around data augmentation.
Just to add the idea here is that similar to Pipeline.gather
we can express data augmentation as something like
val data = CifarLoader(trainLocation)
val featurePipeline = Pipeline.concat {
RandomPatch() andThen
RandomFlip() andThen
Convolver andThen
Pooler
...
}
// Here the RDD that is passed in to the next stage of the pipeline has the union
val solvePipeline = featurePipeline andThen BlockWeightedSolver()