increasing mg sub-cycles crashes with -microphys mg3 -phys cam6

[adamrher]

What happened?

Model crashes between 7 and 14 days when running micro_mg_num_steps= 3,4 with graupel and hail turned on (mg3) and -phys cam6. This configuration appears to be stable for -phys cam_dev, suggesting that the problem is only when combining old pumas (src/physics/pumas-frozen) + mg3 + micro_mg_num_stesp=3,4. Weirdly micro_mg_num_stesp=2 seems to be stable.

The error messages are different depending on the run, but they all exhibit the hallmarks of a blow up (NaN's sent to the land, NaN's in clubb).

I'm uncertain the priority of this issue -- but will add updates if/when I learn more.

What are the steps to reproduce the bug?

see above

What CAM tag were you using?

cam6_3_067

What machine were you running CAM on?

CISL machine (e.g. cheyenne)

What compiler were you using?

Intel

Path to a case directory, if applicable

/glade/scratch/aherring/cam6_3_067_F2000climo_ne30pg3_ne30pg3_mg17_L58_nuopc_1800pes_220715_4mg3steps

Will you be addressing this bug yourself?

No

Extra info

@renerwijn