learning_from_play icon indicating copy to clipboard operation
learning_from_play copied to clipboard

Published 20 hours ago • verified publisher icon sholtodouglas

Dataloader optimisation

Open 0xtristan opened this issue 3 years ago • 0 comments

  • mapping transform BEFORE windowing
  • var seq lens and goal tiling are computationally expensive (we ought to do on device) so I've removed from dataloader
  • multi-threaded tfrecords mapper (NOT the IO, this has to be single threaded)
  • action validation also taxing, it could be risky removing but it's also too slow. We should validate our data once off beforehand

0xtristan avatar Mar 07 '21 22:03 0xtristan