Finish getting hi-res ocean diags working

[bertinia]

Using a case from Justin Small, was able to get 10 years worth of ocean hi-res data through the pyAverager. The diags are still dying because of a missing hi-res phc RHO observation file - and maybe some other files as well.

The incomplete model vs. obs are included here: http://webext.cgd.ucar.edu/BRCP/BRCP85C5CN_ne120_t12_pop62.c13b17.asdphys.001/index.20161004-115725.html

For the timeseries, created the horizontal zonal average files in: /glade/p/cesm/omwg/timeseries_obs_tx0.1v2_62lev

Steps that are needed to finish this:

1. make sure all the hi-res obs files with 62 levels are in the omwg/obs_data
2. rename the timeseries zonal average to include the res and levels in the file name.
3. modify the call to the pyAverager in ocn_avg_generator.py for timeseries so the suffix matches the horizontal za file names from 2.
4. rerun the diags all the way through for model vs. obs and timeseries
5. add an ocn hi-res examples dir
6. make sure the input data repo has all the new files included.

[bertinia]

Here's an update:

1. The pyAverager needs to be updated to handle 62 vertical levels. It currently only works with 60. Sheri is aware and has this on her to-do list.
2. The max number of years that the pyAverager can handle is 10. This is due to a memory limitation and one of the underlying python libraries. Sheri is aware and I think the CISL-ASAP group is looking into alternatives so we can get more years through. The fallback is to use the limited parallelism of the swift implementation and the NCO operators.
3. Ernesto created a SVN branch of the popdiags specifically for hi-res data. His NCL code needs to be merged into the CESM_postprocessing repo version. I can do this but it looks like quite a bit of time/work that I'm not sure is top priority for me with all my tasks associated with getting the release out and the CMIP6 workflow. I can help someone setup a git sandbox if they want to take this on? Ernesto's code is in: https://svn-ccsm-models.cgd.ucar.edu/tools/proc_ocn/pop_diag/branches/popdiag_munoz/
4. We created some hi-res horizontal mean files starting with the PHC hi-res SALT and TEMP obs-data for each basin to be used with the timeseries diagnostics. These files are here: /glade/p/cesm/omwg/timeseries_obs_tx0.1v2_62lev They need to be renamed so they are unique from the corresponding files in the /glade/p/cesm/omwg/timeseries_obs which are associated with gx1v6, 60lev and then the model_timeseries.py code needs to be updated to link the correct files based on grid and vertical levels specified in the XML. I will do this step.
5. We're missing the PHC[2 or3]_RHO0_tx0.1v2.nc needed for the Mixed Layer Depth plots. Dave, it looks like you used some input RHO obs file to generate this plot but I don't know where that file is on glade. http://webext.cgd.ucar.edu/B2000/ocn/hybrid_v5_rel04_BC5_ne120_t12_pop62/mld2.gif Is this the only missing obs data file or are there others? Can we gather them all into the /glade/p/cesm/omwg/obs_data common location?

[bertinia]

The memory bottle-neck for ocean hi-res is related to the creation of the mavg file which contains all the monthly averages for a user specified number of variables.

id="OCNDIAG_PYAVG_MODELCASE_VARLIST"

value="TEMP,SALT,PD,MOC,N_HEAT,N_SALT,UVEL,VVEL,WVEL,SU,SV,IAGE,KAPPA_ISOP,KAPPA_THIC,TAUX,TAUY,SSH,UISOP,VISOP,WISOP,HMXL,HBLT,BSF,TLT,INT_DEPTH,DIA_DEPTH,SFWF,PREC_F,MELT_F,MELTH_F,SHF,SHF_QSW,SENH_F,QFLUX,SNOW_F,SALT_F,EVAP_F,ROFF_F,LWUP_F,LWDN_F"

The pyAverager first computes the monthly average for each variable and each month and writes to an intermediate file. Examples:

• January - BRCP85C5CN_ne120_t12_pop62.c13b17.asdphys.001.pop.h.2006-2015._01_climo.nc
• February - BRCP85C5CN_ne120_t12_pop62.c13b17.asdphys.001.pop.h.2006-2015._02_climo.nc

Etc...

and then the mavg.2006-2015 file is a concatenation of these monthly files.

The NCL scripts assign the following variables to the mavg file:

• SEASAVGTEMP
• SEASAVGFILE
• SEASAVGSALT
• SEASAVGRHO

And the NCL scripts that use these file variables include:

• compute_rho.ncl and compute_rho_diff.ncl
• sst_eq_pac_seasonal_cycle.ncl and sst_eq_pac_seasonal_cycle_diff.ncl
• mld.ncl and mld_diff.ncl

One option to help get around the bottle neck is to by-pass the creation of the mavg file and then modify the NCL scripts to loop over the monthly files. Another option is to generate a mavg file with just SALT and TEMP variables and not all the variables. This second option would be the least intrusive and mirrors what the old gen_mavg_annmean.csh script does.

@bertinia will try running a test with the second option to see if there are any problems with this approach.