nowcasting_dataset

nowcasting_dataset copied to clipboard

Backfill data from 2016 to 2020

1

Let's do this in early 2022 (for WP2). Related to #81

JackKelly

## Detailed Description I'm pretty certain that modern video compression (e.g. AV1) and/or image compression (e.g. AVIF) will result in _much_ smaller file sizes for our image data sources (satellite...

JackKelly

## Detailed Description Download pv output.org dataset for 2020 and 2021 ## Context This will allow #562 to be tested properly

peterdudfield

## Detailed Description Remove renaming of x and y coorindate once 1. https://github.com/openclimatefix/Satip/issues/42 2. https://github.com/openclimatefix/nwp/issues/9 ## Context good to clear up code

peterdudfield

`PVDataSource` is ignoring 52 PV systems from the PassivSystems data... let's manually check if those 52 PV systems really should be removed :slightly_smiling_face:

1

Not urgent ``` 2021-11-17 17:55:06,391 DEBUG processID=1555225 Creating pv DataSource object. | /home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/manager.py#L103 2021-11-17 17:55:06,393 DEBUG processID=1555225 Loading PV metadata from /mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/PV/Passiv/ocf_formatted/v0/system_metadata_OCF_ONLY.csv | /home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/data_sources/pv/pv_data_source.py#L72 DEBUG:nowcasting_dataset.data_sources.pv.pv_data_source:Loading PV metadata from /mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/PV/Passiv/ocf_formatted/v0/system_metadata_OCF_ONLY.csv 2021-11-17...

JackKelly

After `rioxarray` 1.3 is released, remove `OpenData` from `TopographicDataSource.__post_init__()`

2

Right now, `rioxarray` can't open data directly from cloud storage buckets, so `TopographicDataSource.__post_init__()` downloads files to the VM's local hard disk using `OpenData`. But `rioxarray` v1.3 will enable `rioxarray` to...

JackKelly

Start collecting intra-day Sheffield Solar PV Live Regional (GSP-level)

3

## Detailed Description See https://github.com/openclimatefix/predict_pv_yield/issues/90 for more context

JackKelly

## Detailed Description Make sure x and y center from metadata are aligned with GSP id ## Context Useful for fake data to represent real data ## Possible Implementation Options...

peterdudfield

Add GSP shape to dataset

1

When the model is predicting PV yield for an entire Grid Supply Point region, it may be useful for the model to get a binary mask showing the geographical shape...

JackKelly

## Detailed Description - Move imports to top of file - solve circular imports issue ## Context https://github.com/openclimatefix/nowcasting_dataset/pull/314#discussion_r762892363 code is [here](https://github.com/openclimatefix/nowcasting_dataset/blob/jacob/optical-flow-datasource/nowcasting_dataset/data_sources/optical_flow/optical_flow_data_source.py#L73)

peterdudfield