Applying BitInformation to compress WRF model results

[rsignell-usgs]

As mentioned by @rkouznetsov in @https://github.com/milankl/BitInformation.jl/issues/25#issuecomment-998113627, he has applied the BitInformation approach to compressing the WRF model, using this NCKS script:

#!/bin/bash
infile=$1
outfile=$2

list3d='U,V,T,QVAPOR,QCLOUD,QRAIN,QICE,QSNOW,QGRAUP,CLDFRA'

prc3d="--ppc U,V,T=.3 --ppc QVAPOR,QCLOUD,QRAIN,QICE,QSNOW,QGRAUP=.7 --ppc CLDFRA=.3"
  list2d='Q2,T2,PSFC,U10,V10,TSLB,SMOIS,SEAICE,SNOW,SNOWH,SSTSK,LAI,TSK,RAINC,RAINNC,SWDOWN,SWDNB,ALBEDO,UST,PBLH,HFX,QFX,ACHFX,SST'

prc2d='--ppc U10,V10,T2,SMOIS,SEAICE,SSTSK,LAI,TSK,RAINC,RAINNC,ALBEDO,SST,TSLB=.2 --ppc Q2,PBLH,UST=2'
  gridvars="SINALPHA,COSALPHA,MAPFAC_MX,MAPFAC_MY,XLONG,XLAT,XLONG_U,XLAT_U,XLONG_V,XLAT_V,MAPFAC_UX,MAPFAC_UY,MAPFAC_VX,MAPFAC_VY"
  othervars='XLON.*,XLAT.*,Times,ZNU,ZNW,ZS,DZS,HFX_FORCE,LH_FORCE,TSK_FORCE,HFX_FORCE_TEND,LH_FORCE_TEND,TSK_FORCE_TEND,FNM,FNP,RDNW,RDN,DNW,DN,CFN,CFN1,THIS_IS_AN_IDEAL_RUN,RDX,RDY,RESM,ZETATOP,CF1,CF2,CF3,ITIMESTEP,XTIME,P_TOP,T00,P00,TLP,TISO,TLP_STRAT,P_STRAT,SAVE_TOPO_FROM_REAL,ISEEDARR_SPPT,ISEEDARR_SKEBS,ISEEDARR_RAND_PERTURB,ISEEDARRAY_SPP_CONV,ISEEDARRAY_SPP_PBL,ISEEDARRAY_SPP_LSM,C1H,C2H,C1F,C2F,C3H,C4H,C3F,C4F'

ncks -4 -L 5 --baa=5 -v $gridvars,$othervars,$list3d,$list2d $prc3d $prc2d --cnk_dmn bottom_top,1 $infile $outfile

[rsignell-usgs]

@rkouznetsov, would you also be willing to share the code you used to analyze the WRF output to determine the keepbits?

[rkouznetsov]

It is not a result of any derivation. Rather some ad-hoc parameters that, allow for a decent compression, while not affecting SILAM simulations. Note that most of the variables are rounded to keep absolute precision, not relative. E.g. ".7" means that the absolute error will be 1e-7 (kg/kg for specific humidity). As I wrote, the trimming seems to be quite conservative.

[rsignell-usgs]

@rkouznetsov , I'm curious: did you use the BitInformation method to help guide your selection of any of the parameters you used with your NCO/WRF script above (e.g. for variables you were unsure of)? Or did you just choose them on the basis of intuition/experience and negligible changes in model-data comparison skill?

[rkouznetsov]

did you use the BitInformation method

No. It is a script of couple of years ago. The basis is just (subjective) common sense, some idea of sensitivity of SILAM to input uncertainties and some back-envelope estimates. I believe, one could do a better job, but for my purposes the above parameters were "good enough".

Your mileage might differ though...