benchmarks icon indicating copy to clipboard operation
benchmarks copied to clipboard

Published 20 hours ago • verified publisher icon tensorflow

How to split dataset by multi workers?

Open anpark opened this issue 6 years ago • 2 comments

HI, if i have 5 parts input dataset in hdfs, then if i use 5 workers to train 2 epochs i think worker 0 read part-0 2epochs, worker 1 read part-1 2 epochs,..... is it right, if yes, how to do that inside? I don't see dataset has any shard op now.

anpark avatar Apr 23 '18 06:04 anpark

#151 same problem, shift_ratio not used by use_dataset

anpark avatar Apr 23 '18 14:04 anpark

/CC @rohan100jain can you implement shift_ratio with datasets?

reedwm avatar Apr 23 '18 16:04 reedwm