Modify `NonhydrostaticModel` to include different advection schemes for velocities and tracers

[xkykai]

This PR would allow the user to choose the advection schemes they wish to use for velocities and tracers separately. This improves customizability as well as allowing the user to run other arbitrary sets of equations that requires timestepping.

Note: in writing validate_momentum_advection for the NonhydrostaticModel I've taken it from HydrostaticFreeSurfaceModels: https://github.com/CliMA/Oceananigans.jl/blob/1f9bf600ae80fe25477b27536275488708a9ec98/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_model.jl#L213 Is there a potential issue here? Should I move this to Models instead, if validate_momentum_advection is universal?

closes #3332

[simone-silvestri]

Even if curvilinear grids are not supported yet for the non-hydrostatic model, I like the idea of moving validate_momentum_advection to Models like validate_tracer_advection.

[xkykai]

Even if curvilinear grids are not supported yet for the non-hydrostatic model, I like the idea of moving validate_momentum_advection to Models like validate_tracer_advection.

does this mean that I should actually write separate versions of validate_momentum_advection for HydrostaticFreeSurfaceModels and 'NonhydrostaticModel` instead?

[simone-silvestri]

Even if curvilinear grids are not supported yet for the non-hydrostatic model, I like the idea of moving validate_momentum_advection to Models like validate_tracer_advection.

does this mean that I should actually write separate versions of validate_momentum_advection for HydrostaticFreeSurfaceModels and 'NonhydrostaticModel` instead?

Just moving validate_momentum_advection to the Models module is fine. In the end, validate_momentum_advection only depends on the grid and the advection scheme.

[simone-silvestri]

This is a breaking change in user API so we need to bump the major version (in the Project.toml file) and change the examples to reflect the change.

[xkykai]

I can change all instances in ./examples. Should I change everything else too? If not which ones should I change?

[glwagner]

Shouldn't we change everything? The things we don't change won't work, right? Are we stil planning to support the advection kwarg, or is it required that we separate momentum_advection and tracer_advection ?

[glwagner]

It might help to write up in detail what the proposed API change is into the original post at the top of the PR, and give a few examples of new + old usages