Bittree updates

[akashdhruv]

Summary

This PR introduces updates to regridding using Bittree

Additional background

We have introduced a new feature to use AMR in Octree mode. This was a result of ongoing performance studies which identified that Octree mode using Bittree scales better than AMReX native patch-based implementation.

Checklist

The proposed changes:

• [ ] fix a bug or incorrect behavior in AMReX
• [x] add new capabilities to AMReX
• [ ] changes answers in the test suite to more than roundoff level
• [ ] are likely to significantly affect the results of downstream AMReX users
• [x] include documentation in the code and/or rst files, if appropriate

[akashdhruv]

Currently performing tests. This PR is created to enable discussion as we tweak the code and get it ready for production.

[rmrsk]

I'm curious what is meant by 'octree-mode' since octrees have a regular/leaf node separation where potentially nothing is allocated for regular nodes. But that's (almost) never the case for patch-based AMR...

[akashdhruv]

In the context of this PR octree mode refers to an octree refinement using the patch based algorithm. The idea is to be able to do regridding in this configuration using Bittree and bypass the native MakeNewGrids. Bittree has shown performance improvement for PARAMESH for Flash-X and we want to see if it can improve regridding for AMReX

[tklos96]

@WeiqunZhang I made various optimization changes and updated the passing of pointers as you suggested. I have one last problem before it's ready to merge - Subcycling. When I turn subcycling on for the Advection test, I run into issues after ~35 timesteps - the solution blows up and the grid layout becomes unpredictable. I am wondering if the Bittree logic doesn't properly account for the lbase or something. I can't tell what's happening though, any suggestions?

[WeiqunZhang]

We need to introduce USE_BITTREE into GNUMake system and add -DAMREX_USE_BITTREE to the compiler flags. Then we can use ifdef AMREX_USE_BITTREE on the bittree code that has extra dependence.

[WeiqunZhang]

For the failed subcycling test, is it 2d or 3d?

[tklos96]

It was 2d.

[akashdhruv]

We need to introduce USE_BITTREE into GNUMake system and add -DAMREX_USE_BITTREE to the compiler flags. Then we can use ifdef AMREX_USE_BITTREE on the bittree code that has extra dependence.

This is done. Also created a separate directories Src/Extern/Bittree and Tools/GNUmake/packages/Make.bittree to manage source code and dependencies.

[akashdhruv]

./configure --enable-bittree=yes is the new option to build AMReX in Bittree mode. Default is no. Environment variable BITTREE_$(DIM)D_HOME should be defined to store Bittree library path. $(DIM) is the dimension

[akashdhruv]

Additional task: Setup a test in AMReX test suite @WeiqunZhang can you do that on your end?

[WeiqunZhang]

Yes, once the bittree repo is public, I will set up a regression test.

[WeiqunZhang]

The last time I tried, I could not reproduce the blow-up issue with subcycling. After the two minor comments are addressed, I think we can merge this first, because this is a big PR.

[akashdhruv]

I found an issue with a 3D setup in Flash-X. Give me a few days to resolve that and then we can merge

[akashdhruv]

Performance of Paramesh vs AMReX in Flash-X

[akashdhruv]

Continued in #3547