Logging variable values outside of valid ranges in GPU kernels

[charleskawczynski]

We need a way to log (exactly once) variable values found outside of their valid ranges, and this needs to be GPU compatible. Right now, we don't print anything, although we could print using KernelAbstractions (here). However, this is will bloat the log and is not really helpful.

Over-arching issue here

[trontrytel]

Going to paste my notes here and use the issue to track things :)

Think if we can use any of the ideas here:

• https://github.com/CliMA/Thermodynamics.jl/pull/158
• https://github.com/CliMA/Thermodynamics.jl/blob/main/src/DataCollection.jl

to be able to preserve information on when the model is trying to call parameterizations outside of their allowed range. We can't throw warnings on GPUs. Plus, polluting the screen with massive numbers of warnings is not useful in a big simulation anyway.

Maybe instead we could create some sort of a log file with information on:

• what function was attempted to be called outside of the allowed range
• how many times
• with verbose option to also save the inputs?

The only thing is that CM.jl is just a library of functions. So such a thing might need to be implemented on the side of the model that is using it? Or would we still need to provide a logger inside CM.jl?

The list of functions we would need it for (we might need to add one or two more):

• https://github.com/CliMA/CloudMicrophysics.jl/blob/60cfdb81bb600ee7608b0ff0f1b2f4569876fa87/src/Common.jl#L134
• https://github.com/CliMA/CloudMicrophysics.jl/blob/60cfdb81bb600ee7608b0ff0f1b2f4569876fa87/src/IceNucleation.jl#L45
• https://github.com/CliMA/CloudMicrophysics.jl/blob/60cfdb81bb600ee7608b0ff0f1b2f4569876fa87/src/IceNucleation.jl#L120