FNO and RF crash with default Bomex parameters

[charleskawczynski]

I recently found, looking through our recent builds, that FNO and RF entrainment schemes are crashing with the default Bomex parameters. This is because we exit out before performing tests after running these cases (since we don't have regression data). I think a quick "fix" would be to move some of the sanity tests before exiting out (e.g., check for NaNs, check that the error code is :succeed). We could go a step further and add regression tests against main, but that would require more time than I have at the moment.

@zongyi-li, can you update the parameters in a PR so that these entrainment schemes don't break? Once you do I can rearrange the tests to hopefully avoid this breaking by default behavior.

[charleskawczynski]

cc @zongyi-li, @mattlevine22

[charleskawczynski]

Also, because these tests break, the stack trace is dumped and that seems to take a long time on CI. So, our CI time is currently limited by our RF test.

[mattlevine22]

cc @nickhnelsen

[mattlevine22]

Is this fixed yet?

[charleskawczynski]

Here are some parameters for RF:

"turbulent_entrainment_factor": 0.1020095424633089
"tke_ed_coeff": 0.15819768881242882,
"tke_diss_coeff": 0.05314798799348995,
"static_stab_coeff": 0.5503981488289578,
"tke_surf_scale": 2.073978192805755,
"Prandtl_number_0": 0.9466537497932478,
"pressure_normalmode_adv_coeff": 0.0007001798770002688, # momentum exchange parameters
"pressure_normalmode_buoy_coeff1": 0.08019416893477578,
"pressure_normalmode_drag_coeff": 25.17739666798333,
"surface_area": 0.10993739346876041, # surface