nowcasting_dataset icon indicating copy to clipboard operation
nowcasting_dataset copied to clipboard

Experiment with calling `dataset.load()` _after_ joining examples into batch

Open JackKelly opened this issue 3 years ago • 0 comments

For the Zarr DataSources, it may be faster to load the data into memory after joining (lazily loaded) examples.

i.e. call .load() towards the end of get_batch() instead of at the end of get_example().

This should allow dask to do a better job of scheduling what needs to be done. And might result in faster times per batch.

JackKelly avatar Nov 18 '21 19:11 JackKelly