SILHS subcolumns output as all zeros

[bstephens82]

What happened?

When running SILHS with either -phys cam6 or -phys cam_dev (although for cam_dev see the necessary code changes associated with issue #844), SILHS subcolumn fields are being output to netcdf as all zeros, despite being nonzero right before the outfld call and even within the outfld call. I have also tried setting the entire SILHS subcolumn field WM_lh_out = 1.0_r8 immediately before the outfld call in subcol_SILHS.F90, just to make sure the outfld call is not receiving anything crazy, and it still outputs all zeros in the netcdf history file.

What are the steps to reproduce the bug?

• Using a recent tag (I've tested up to cam6_3_155), set up a CAM test case with "-silhs -psubcols 4" included in CAM_CONFIG_OPTS. You can use either -phys cam6 or -phys cam_dev (although cam_dev will require some code changes to cam_dev/micro_pumas_cam.F90 associated with separate issue #844 to work). Ask @bstephens82 for the tentative code changes (or see PR #1016).
• Then add SILHS output fields to fincl1 in user_nl_cam, for example fincl1 = 'SILHS_WM_SCOL' or 'SILHS_THLM_SCOL'.
• You will also need to add use_gw_convect_dp=.false. to your namelist since turning on SILHS will automatically set deep_scheme='off' and for some reason that won't run with the gw flag set to T. This seems like a separate bug to me, but in any case this is necessary when deep_scheme='off'.
• I believe you also need to SourceMod in cam/subcol_SILHS.F90 where you change l_outfld_subcol=.true. (this is hard-coded to false by default in that module...I am not sure why).

What CAM tag were you using?

cam6_3_155

What machine were you running CAM on?

CISL machine (e.g. cheyenne)

What compiler were you using?

Intel

Path to a case directory, if applicable

/glade/derecho/scratch/stepheba/cam155_0/

Will you be addressing this bug yourself?

Any CAM SE can do this

Extra info

I spoke to @brian-eaton about this and he suggested this might be related to something similar we are seeing in COSP.

Worth knowing: There is an automated test that runs SILHS when a new tag is created, thanks @Katetc for telling me about this test. It can be run with

./create_test SMS_Ln9.f19_f19.F2000climo.derecho_intel.cam-silhs

However, this test runs with -phys cam6, so it will not catch the cam_dev-SILHS bug (#844) and, while the namelist that it automatically generates does include some SILHS output fields (namely SILHS_CLUBB_PRECIP_FRAC and SILHS_CLUBB_ICE_SS_FRAC), these are included in fincl1 and the first integer in nhtfrq is 0 by default, meaning monthly output. Since this test only runs for a few time steps, it does not appear that it will actually output any SILHS fields, so it would have no way of catching this error where the output is all zeros. I checked all the output files after running the test, to verify there was no SILHS output. Then I set nhtfrq=1 for the h0 output files, and the SILHS output was all zeros.

I have also tested this problem with finite volume dycore in addition to the SE dycore. That has no effect on the zeros (I was wondering whether it might've had something to do with interpolation).