nowcasting_dataset

nowcasting_dataset copied to clipboard

Check SatelliteDataSource and NWPDataSource put x and y dims in the same order

2

I have a feeling that NWP puts the dims in this order: y, x. Whilst satellite puts the dims in order x, y. Even if the dims are in different...

JackKelly

Write tests for all the `Manager` methods

3

JackKelly

Run validation script at the end of `prepare_ml_data.py`?

2

## Detailed Description Maybe we should _always_ validated the on-disk batches? (Let's wait for PR #300 to be merged before working on this)

JackKelly

Not super-urgent. For now we can just use the same variable names we've used in the past.

JackKelly

## Detailed Description Include the forecast horizon of each NWP timestep in the output batches. ## Context As a general rule, NWPs get less skillful the longer the forecast horizon....

JackKelly

## Detailed Description At the moment, we use NWPs with an init time at or just before the start datetime of the historical data for each example. Instead, we should...

JackKelly

Method to drop 'padded' pv systems

2

## Detailed Description Would be useful to have a method to drop 'padded' pv systems. These are padding out zero so that the dataset can be save in an efficient...

peterdudfield

Remove pytorch from `scripts/validate_ml_data.py`

1

Related issues: - #138 - #86

JackKelly

Move our data download & conversion scripts into separate repos (preparing for production)

5

Our code for downloading and converting satellite data lives in the Satip repo. Our code for downloading and converting other data sources (NWP, GSP PV, etc.) lives in this repo....

JackKelly

Use smaller dtypes for saved data

11

int16 for both NWP & satellite? Maybe even uint8 (and use 0 for NaN???)

JackKelly