[WIP] Preconditioning CG solver with multigrid

[elise-palethorpe]

Using multigrid as a preconditioner to the congujacte gradient solver

cc @navidcy

closes #2655

[glwagner]

Just a minor comment to use nouns for file names (and as descriptive as possible), ie "multigrid_preconditioner" rather than "testing_preconditioner"

[navidcy]

@elise-palethorpe, MGImplicitFreeSurfaceSolver needed to be defined first before used as a preconditioned method in PGC. I flipped the order of the include files.

[navidcy]

@elise-palethorpe also perhaps change the validation script name as @glwagner suggest?

[navidcy]

@glwagner would you be keen to review this?

[elise-palethorpe]

@elise-palethorpe also perhaps change the validation script name as @glwagner suggest?

Do we need the file at all? Can I just include mg preconditioner in doubly_bounded_poisson.jl

[navidcy]

@elise-palethorpe also perhaps change the validation script name as @glwagner suggest?

Do we need the file at all? Can I just include mg preconditioner in doubly_bounded_poisson.jl

Yes, do that!

[navidcy]

@elise-palethorpe, I added some proper bench tests in doubly_bounded_poisson.jl.

Seems like the PCG solve! has many allocations! I don't know why... @glwagner, @simone-silvestri?

[navidcy]

I used Julia v1.8 memory allocation profiler and I saw that this https://github.com/CliMA/Oceananigans.jl/blob/a5409c27380a205f27961cc6ce4f24581b98fab8/src/Fields/field.jl#L500 creates a bunch of allocations.

[simone-silvestri]

Also, maybe we want to have a dot product that allows masking immersed regions

[navidcy]

Some benches.

julia> include("validation/elliptic_solvers/doubly_bounded_poisson.jl")

julia> include("doubly_bounded_poisson.jl")
[ Info: Solving the Poisson equation with an FFT-based solver...
  123.083 μs (93 allocations: 17.56 KiB)
[ Info: Solving the Poisson equation with a conjugate gradient iterative solver...
  64.748 ms (80482 allocations: 25.93 MiB)
[ Info: Solving the Poisson equation with the Algebraic Multigrid solver...
  9.491 ms (498 allocations: 8.46 MiB)
[ Info: Solving the Poisson equation with a conjugate gradient preconditioned iterative solver w/ AMG as preconditioner...
  47.891 ms (12771 allocations: 111.97 MiB)

(The above are results with @btime... It's pretty obvious than PCG brings allocations to the roof...)

[navidcy]

ok @glwagner this is ready to review!

A question for you: We didn't include fill_halos!(z) in the MG-preconditioning:

https://github.com/CliMA/Oceananigans.jl/blob/bd0363a54c6f0054cc7a67f5f61e30f12a839bc1/src/Models/HydrostaticFreeSurfaceModels/pcg_implicit_free_surface_solver.jl#L231-L235

Do you think we should have had?