nowcasting_dataset icon indicating copy to clipboard operation
nowcasting_dataset copied to clipboard

Allow different size regions of interest for history and forecast

Open JackKelly opened this issue 3 years ago • 3 comments
trafficstars

Detailed Description

Implement a config option to specify different sized regions of interest for the history versus the forecast.

Context

The MetNet papers used much larger regions of interest for the history than for the forecast. We should probably do the same :slightly_smiling_face:

JackKelly avatar Feb 16 '22 10:02 JackKelly

This is a good point, obvious to see in the UCL ML competition, where a bigger input image would have made better results

Perhaps in Satellite we switch to two datavars

  • data_historic, save images of size X
  • data_future, saves images of size Y

peterdudfield avatar Apr 08 '22 07:04 peterdudfield

I'm by no means certain but, in order for the XArray coords to work correctly, I think we might have to save two separate sets of NetCDF files: one set for historic data, and a different set for future data. Which is maybe getting complicated! The only way I can think of putting two different size images into a single NetCDF file would be to have two sets of coords, too (e.g. a x_osgb_future and an x_osgb_historic) which would break all our existing code that reads from the x_osgb coords.

Maybe, for the time being, we can keep things simple and use the large size for both historic and future data in a single NetCDF file and see how insane the file sizes get :slightly_smiling_face:

JackKelly avatar Apr 09 '22 12:04 JackKelly

Yea, for the moment let's do a single set.

Probably is the right way to do it, split it up with two different netcdf files. This also reduces the chance of 'data leakage', i.e using the future to help the model

peterdudfield avatar Apr 11 '22 07:04 peterdudfield