Replace monopole gravity with Poisson gravity?

[maxpkatz]

Even though in principle Poisson gravity can be used in the places where we currently use monopole gravity, we use monopole gravity only in 1D spherical coordinates (and can use it in 2D cylindrical and 3D Cartesian for spherically symmetric mass distributions). The first Castro paper observes that at low resolution monopole is slightly better than Poisson for the StarGrav problem (though the difference converges away; see Figures 10 and 11), and also that monopole scaled much better than Poisson on CPUs (see Figure 2).

However, maintaining monopole gravity comes at a significant cost in code complexity (and I'm not fully convinced it is actually correct with AMR). It is also difficult to parallelize the monopole algorithm effectively. So we should consider dropping monopole gravity and using Poisson gravity in its place. That way we can straightforwardly accelerate these calculations on GPUs. On today's supercomputers using GPUs we can easily achieve the resolutions at which Poisson and monopole yield similar accuracy, so the only question is performance. The scaling results in Figure 2 of the Castro paper make a compelling case for monopole gravity on CPUs, but I don't think they are relevant for modern heterogeneous architectures. It's possible that if we ported monopole gravity to GPUs it would still beat Poisson gravity in performance, but the reward for that effort is low compared to just using Poisson gravity always.

[maxpkatz]

@asalmgren a penny for your thoughts

[zingale]

this would address #92