Train on GFS Data

[jacobbieker]

Detailed Description

GFS data is publicly accessible and so we can make that dataset open, as well as its one of the ones used in the original paper, although with 1 degree data, not the 0.25 degree data I want to try with.

Context

Possible Implementation

[jacobbieker]

Data is being pushed to here for training: https://huggingface.co/datasets/openclimatefix/gfs-reforecast which contains/will conatin raw observations, processed observations, GFS operational forecasts, and FNL Analysis files on 0.25 degree grids for 2016-2022. As well as the GEFS reforecasts from 2010-2019, and potentially back to 2000. Everything is in Zarr format, as it compressed a lot better, although it does mean that the data files all need to be decompressed on download.

[jacobbieker]

Also, just got a model training on 0.25 degree analysis file data for 2016-01-01. It learns things and fits on an 8GB 2070 laptop GPU if downsampling to 1 degree resolution.