Peter Dudfield

Results 255 comments of Peter Dudfield
trafficstars

Discussion: Should we be using random numbers in our tests?

Perhaps 'seeding' them is a good comprise?

Run validation script at the end of `prepare_ml_data.py`?

The validation script is 'abit' / 'a lot' out of data. Itll need some work to update. The good thing is the Batch validates each data source as we go....

Experiment with better compression for on-disk batches

I remember using something like 'gzip' made the files smaller. But then it took longer to load. http://xarray.pydata.org/en/stable/generated/xarray.Dataset.to_netcdf.html I'm not sure on the right balance here also just some general...

Use smaller dtypes for saved data

Might as well - small amount of work, for small win

Use smaller dtypes for saved data

Moving back here > > > Do you mind expanding on what needs doing for the NWP data? > > > > > > Sure! > > The short answer...

Use smaller dtypes for saved data

Would be interesting to know if some default compression would be the same as doing this rescaling (see above). Probably only way it to try it out ...

Use smaller dtypes for saved data

I was just about to type the same, so low hanging fruit, change it to float16 the test data was float32 = 13M float16 = 4.6M int16 = 3 M...

Use smaller dtypes for saved data

Ive pushed that now to PR #335

Use smaller dtypes for saved data

Ill keep it open, incase we do want to go down the int8/int16 route

Remove pytorch from `scripts/validate_ml_data.py`

I think needs a rewrite of the validation script too Potentially something we can live we out as the 'manager.py' now does some validation