nowcasting_dataset icon indicating copy to clipboard operation
nowcasting_dataset copied to clipboard

Compress NWP and satellite batches using AV1 or AVIF

Open JackKelly opened this issue 3 years ago • 0 comments

Detailed Description

I'm pretty certain that modern video compression (e.g. AV1) and/or image compression (e.g. AVIF) will result in much smaller file sizes for our image data sources (satellite and NWP).

Context

As we start to experiment with examples with larger geospatial extents, and as we train on a variety of environments (on-prem, GCP, AWS, Lancium), it'll become increasingly important to efficiently compress imagery.

Possible Implementation

See https://github.com/openclimatefix/Satip/issues/45 for some notes

We could save each example as a separate video file (e.g. by outputting TIFFs from Python and using ffmpeg to encode as AVIF). Or use one video file per batch where all the examples are appended into a single video sequence.

It might be slow to load 32 separate video files for each batch. So we could also try to use AVIF or AV1 within a NetCDF file.

JackKelly avatar Jan 07 '22 15:01 JackKelly