Derefine doesn't necessarily derefine

[drummerdoc]

Pele codes need to avoid EB intersections with any coarse fine boundaries. So, either the EB is completely refined to the finest AMR level (which can get unnecessarily expensive) or not, but then the user has to try to ensure this using refinement tagging control. While it's straightforward to "untag" cut cells above a specified level, new boxes usually contain some untagged cells (a la grid_eff). We have cases where the cells in new boxes (usually the corners of the new boxes) can reach out over the EB, thus generating coarse-fine/EB intersections that crash the code. We are thinking about complex/invasive hacks that involve complementing the generated grids while still in blocking_factor coordinates with a special EB-following box array. Wondering if anyone else has already solved this issue though in a more clever way.

[rmrsk]

I'm curious why EBxCF needs to be avoided...? It sounds like supporting EBxCF would be the preferred solution (?)

So I haven't throught this completely through, but one potentially simple solution that I can think of is to "untag" cells up to (max_grid_size+err_buff distance) from the EB, and possibly complement this with a "refine the EB to level X" criterion if you want the EB resolved to more than the base level. This should require 1). An MPI gather of all the tags (hello memory bottleneck). 2) Using the IndexSpace for removal of tags "too close" to the EB. 3) A gather-broadcast of removed tags which are then stripped from your final tags just before passing them into the box generator.

But it's also not completely clear to me what an acceptable solution is here. If there are multiple bodies, should they all be at the same resolution, or is that EBxCF needs to be avoided for individual bodies? Or should the EB be generated to the finest resolution that does not create an EBxCF situation? If your tags leads to an EB that is only partially covered by grids, should one fill the remaining holes or remove the patches on that level altogether?

[drummerdoc]

Thanks Robert. First, refining all the EB is simply not doable...creates >100B cell domains or larger for the case we're looking at...to big. Allowing EBxCF requires properly treating re-redistribution. Given all the problems we've had enforcing hard-bounded states with the regular redistribution (mass fractions summing to 1, but remaining strictly between 0 and 1), we've not had the interest to take that one on yet.

The simplest way to see the issue though is to imagine you tag a cell for refinement that is 13 cells from the EB and n_error_buf = 0. Say blocking_factor is 16. Amr could generate a new box that is 16 cells across that covers the tagged cell, but reaches over far enough to intersect the EB. You can't chop away the part of box sticking out that far even if it doesn't cover tagged cells, because you'd violate blocking_factor. You can try to mess with grid_eff (we do that now) but there's still no guarantee we don't hit the problem. You really need Amr to generate a different box that is instead shifted away from the EB such that the tagged cell is more centered in the new box. But we just don't have the kind of control or feedback that would be required to do that.

[rmrsk]

Marc, thanks for the clarifications. I did understand the problem, and the approach I outlined above should be sufficient for untagging the tagged cell that's 13 cells away from the EB, no? You'll avoid tags near the EB and thus do not create an EBxCF situation. But all your grids also end up a distance 'blocking_factor' away from the EB, and if there are multiple bodies they become resolved to same resolution. I don't know if that solution is acceptable though. From your response it sounds like you want to cover those tags as much as possible, and rather chop/remove boxes once they become a problem. But that sounds far from trivial since shifting/chopping/removing problematic boxes could create EBxCF situations elsewhere..

[drummerdoc]

Thanks. I should try it. But it’s the diagonal version of the issue that is actually breaking our case in particular. It might be that accounting for the extra distance off coordinate axis might fix it though. Will try. Thanks for the input.