nowcasting_dataset
nowcasting_dataset copied to clipboard
Compress NWP and satellite batches using AV1 or AVIF
Detailed Description
I'm pretty certain that modern video compression (e.g. AV1) and/or image compression (e.g. AVIF) will result in much smaller file sizes for our image data sources (satellite and NWP).
Context
As we start to experiment with examples with larger geospatial extents, and as we train on a variety of environments (on-prem, GCP, AWS, Lancium), it'll become increasingly important to efficiently compress imagery.
Possible Implementation
See https://github.com/openclimatefix/Satip/issues/45 for some notes
We could save each example as a separate video file (e.g. by outputting TIFFs from Python and using ffmpeg to encode as AVIF). Or use one video file per batch where all the examples are appended into a single video sequence.
It might be slow to load 32 separate video files for each batch. So we could also try to use AVIF or AV1 within a NetCDF file.