Castro icon indicating copy to clipboard operation
Castro copied to clipboard

Implement optimal subcycling

Open maxpkatz opened this issue 6 years ago • 1 comments

Optimal subcycling is done in Nyx, and I want to implement this as an option in Castro. This involves two steps:

  1. during timestep estimation, call the Amr function which determines whether to subcycle the finer levels, and set the timestep appropriately

  2. during the advance, do the hydro advance / reflux and gravity solves on the current level and all finer levels which have ncycle == 1

maxpkatz avatar Dec 12 '17 21:12 maxpkatz

The easiest way to start here would be to implement amr.subcycling_mode = None. In that case, an advance would notionally be:

for (lev = 0; lev <= finest_level; ++lev) {
    do_old_sources();
}
for (lev = 0; lev <= finest_level; ++lev) {
    do_hydro();
}
for (lev = 0; lev <= finest_level; ++lev) {
    do_new_sources;
}

This would then be able to benefit from efficiencies such as skipping the calculation of the sources on zones that are not leaf zones, and from doing a single multi-level gravity solve rather than level-by-level solves. Given the computational cost and complexity of Poisson solves in particular, it seems worth the effort to implement this, as there's plenty of cases where one wants to skip subcycling (particularly in cases where there are refinement boundaries in interesting high-density parts of the domain, as reflux operations tend to be problematic there).

But this is not trivial to do with the level-by-level advance that is currently written. As one example, we return a status code on the advance, but that status is assuming a failure or success on that level, not on some higher level. And we'd have to be very careful that references like S_new are always referring to the MultiFab data from the correct level.

So the first step here is to do a little re-writing of do_advance_ctu() to ensure that it will be straightforward to try out changes that loop over levels.

maxpkatz avatar Jun 04 '23 02:06 maxpkatz