Unphysical microphysics tendencies to grid mean with slight changes in the resolution

[yairchn]

When trying to explore grid stretching I got a breaking simulation in which the grid mean q_tot and θ_liq_ice obtain unphysical values q_tot = -0.4914666150503556 θ_liq_ice = -395.07767804554845 At a pressure level of ~900mb. Printing the tendencies of these variables (and I am not sure f these are the update tendencies or the previous tilmestep tendencies that were used to get these values -- @charleskawczynski ?) tendencies_gm.q_tot = -0.138 tendencies_gm.θ_liq_ice = -909.850 The corresponding microphysical tendencies are:

aux_bulk.qt_tendency_precip_formation = 0.0 aux_en.qt_tendency_precip_formation = -0.1951825073725916 aux_tc.qt_tendency_precip_sinks[k]) = 0.056559807631770376 and

aux_bulk.θ_liq_ice_tendency_precip_formation = 0.0 aux_en.θ_liq_ice_tendency_precip_formation = -469.8729492625373 aux_tc.θ_liq_ice_tendency_precip_sinks = -437.64272447413254.

By changing the grid spacing from 30 to 20 (in the stretched grid) this problem disappears and the simulation looks reasonable. Any suggestions ?

These results can be reproduced by running TRMM in this branch: https://github.com/CliMA/TurbulenceConvection.jl/tree/yc/issues_microphy_src

[charleskawczynski]

Can we make a smaller reproducer? It would be helpful to make a branch off main with a minimal set of changes.

Printing the tendencies of these variables (and I am not sure f these are the update tendencies or the previous tilmestep tendencies that were used to get these values

All tendencies computed are for the update from time t to time t+dt.

By changing the grid spacing from 30 to 20 (in the stretched grid) this problem disappears and the simulation looks reasonable.

Where does this happen in z? Can you also point to where you're printing (on a commit that's merged into main, not a branch, so that it doesn't go stale)?

I think we can trace what's going wrong by inspecting each individual tendency. @trontrytel, up for the task once we have a minimal deviation branch?

[trontrytel]

I can take a look in the new sprint cycle. A minimal deviation branch would be very helpful.

[trontrytel]

Maybe it would be good to test if we get the same error without quadratures and with quadrature type set to gaussian. There seems to be a lot of noise with lognormal quadrature points in Rico. especially if only using 3x3 sampling points.