Satip
Satip copied to clipboard
Published
20 hours ago •
openclimatefix
openclimatefix
Major rewrite
trafficstars
I thought it we are considering a major rewrite, then it might be worth say why? and what functionality we have right now. Re writing the nwp-consumer has been a success.
Why
- make code easier to read
- easier to debug
- easier to expand on
- join together training dataset code and live consumer. Currently training dataset code is in dagster repo (TODO add link)
Requirements
- Get 3 different satellite data, 0, 9 and 45.5.
- For 0 and 9, get both the HRV and non-HRV data
- Optional to scale data, to match exsisting google public dataset data
- Dataset: Make training dataset, and merge to monthly zarrs
- Live: Get live data
- Live: When getting live data, save raw and process timestamp files so that we dont have to repeat this everytime the consumer runs
- Live: Save 'latest' file, so ML models can just load that file
- Live: If 0 degrees fails, pull 9 degree satellite
- Live: Pull 0 degree in 5 minut intervals, pull 9 and 45.5 deg in 15 minute intervals
- Dockerize service, so that live and dataset can use the same docker file
- Live: Save raw data and processed data, so that we only have to pull raw data once
Optional
- Live: For live data, with 9 degrees, use data tailor service (we might find we might not need to). If we do this, we need a way to run the data tailor clean up service too.
- Add options for Goes satellite back in
in here - https://github.com/openclimatefix/satellite-consumer/issues/1