keystone icon indicating copy to clipboard operation
keystone copied to clipboard

Fix partitioning in Image Dataloaders

Open etrain opened this issue 9 years ago • 0 comments

Right now, Image DataLoaders partition on the file names (which works well for ImageNet style collections). While it's possible to repartition manually after images have been loaded, the API indicates that passing a Some(numPartitions) will yield an RDD with that many partitions.

Should be a small change to ImageLoaderUtils.getFilePathsRDD or upstream uses of it.

etrain avatar Aug 19 '15 02:08 etrain