nowcasting_dataset
nowcasting_dataset copied to clipboard
Experiment with calling `dataset.load()` _after_ joining examples into batch
For the Zarr DataSources, it may be faster to load the data into memory after joining (lazily loaded) examples.
i.e. call .load() towards the end of get_batch() instead of at the end of get_example().
This should allow dask to do a better job of scheduling what needs to be done. And might result in faster times per batch.