Castro
Castro copied to clipboard
AMReX-Astro
state_nghost gives different answers
Run reacting_bubble with inputs_2d_test as:
./Castro2d.gnu.MPI.ex inputs_2d_test castro.state_nghost=0 max_step=10
and then
./Castro2d.gnu.MPI.ex inputs_2d_test castro.state_nghost=1 max_step=10
and there are nontrivial differences:
variable name absolute error relative error
(||A - B||) (||A - B||/||A||)
----------------------------------------------------------------------
level = 1
density 0.8425712585E-03 0.3714247577E-12
xmom 320278.7954 0.2100573021E-05
ymom 267636.6177 0.5798493429E-08
zmom 0.000000000 0.000000000
rho_E 0.1145691116E+16 0.2984930291E-12
rho_e 0.1145691116E+16 0.2984930291E-12
Temp 0.5298435688E-01 0.7741032661E-10
rho_C12 0.2527236938E-03 0.3713546910E-12
rho_O16 0.5886554718E-03 0.3707040693E-12
rho_Mg24 0.4141510243E-05 0.8551035646E-06
phiGrav 0.000000000 0.000000000
grav_x 0.000000000 0.000000000
grav_y 0.000000000 0.000000000
grav_z 0.000000000 0.000000000
omegadot_C12 0.1932632789E-11 0.4326742623E-06
omegadot_O16 0.3865265579E-11 2.000000000
omegadot_Mg24 0.9663164850E-11 0.2163371005E-05
enuc 4456347.807 0.1781261133E-05
rho_enuc 0.6370308828E+16 0.1823618222E-05
pressure 0.6506360057E+15 0.4507366116E-12
kineng 6121024640. 0.7837523051E-08
soundspeed 0.5171298981E-03 0.5599428806E-12
Gamma_1 0.1355582313E-11 0.1003833509E-11
MachNumber 0.2027028380E-12 0.5056993624E-08
entropy 0.1647248864E-02 0.2676302374E-10
magvort 0.9423386951E-10 0.1382169941E-07
divu 0.4817392440E-09 0.7182512651E-07
eint_E 629504.0000 0.3720496785E-12
eint_e 629504.0000 0.3720496785E-12
logden 0.1634248292E-12 0.1746787563E-13
StateErr 0.8425712585E-03 0.3714247577E-12
X(C12) 0.1387778781E-14 0.4625929269E-14
X(O16) 0.1887379142E-14 0.2696255917E-14
X(Mg24) 0.2220446049E-14 0.6401404622E-06
x_velocity 0.1606318879E-03 0.1537683917E-05
y_velocity 0.1691162797E-03 0.4977988586E-08
z_velocity 0.000000000 0.000000000
t_sound_t_enuc 0.2166523193E-14 0.1852579462E-05
magvel 0.1692252299E-03 0.4981195562E-08
radvel 0.1713951117E-03 0.8696103057E-08
magmom 267636.6167 0.5798493403E-08
angular_momentum_x 0.000000000 0.000000000
angular_momentum_y 0.000000000 0.000000000
angular_momentum_z 0.3697120700E+14 0.6659200778E-08
maggrav 0.000000000 0.000000000
pi 0.6506360057E+15 0.9382965888E-07
pioverp0 0.4628912277E-12 0.5534889676E-07
rhopert 0.8425712585E-03 0.2109698140E-09
tpert 0.5298435688E-01 0.2100369301E-09
There is a difference here even if all (non-hydro) source terms are turned off. In that case, the only possible explanation for the difference is that the ghost zones in Sborder are different in the hydro, which changes the PPM reconstruction on the edges. So the problem there must be in how we do expand_state.
In expand_state we call clean_state and then FillPatch NUM_GROW zones into Sborder. When the state has no ghost zones, this makes sense. When the state has a ghost zone, the FillPatch will take advantage of the data in that ghost zone rather than extrapolate over it. That is, we extrapolate to zones -4, -3, and -2 using information from zone -1, whereas in the former case we're extrapolating from information in zone 0. It seems like the only way to force consistency here would be to require FillPatch to ignore (and overwrite) data in the FillPatch from the existing ghost zones.
Note that the call to clean_state only works on the valid zones. The above issue is not resolved if we change this so that it works on the ghost zones too -- we'd still have the difference about where the extrapolation is coming from.
the diffs still exist with the current version of the code
variable name absolute error relative error
(||A - B||) (||A - B||/||A||)
----------------------------------------------------------------------
level = 1
density 0.1773834229E-03 0.7819468583E-13
xmom 89649.14879 0.5879709127E-06
ymom 77934.85237 0.1688501079E-08
zmom 0.000000000 0.000000000
rho_E 0.4090183255E+15 0.1065637301E-12
rho_e 0.4090183255E+15 0.1065637301E-12
Temp 0.8660793304E-02 0.1265344864E-10
rho_C12 0.5245208740E-04 0.7707361511E-13
rho_O16 0.1254081726E-03 0.7897543154E-13
rho_Mg24 0.5465881126E-05 0.1128548033E-05
phiGrav 0.000000000 0.000000000
grav_x 0.000000000 0.000000000
grav_y 0.000000000 0.000000000
grav_z 0.000000000 0.000000000
omegadot_C12 0.2898949184E-11 0.6490113934E-06
omegadot_O16 0.3865265579E-11 2.000000000
omegadot_Mg24 0.7730683404E-11 0.1730729978E-05
enuc 4456347.807 0.1781261133E-05
rho_enuc 0.6369866837E+16 0.1823491694E-05
pressure 0.3625639593E+15 0.2511709298E-12
kineng 809891648.0 0.1037006793E-08
soundspeed 0.4492998123E-03 0.4864971685E-12
Gamma_1 0.1270095140E-11 0.9405286896E-12
MachNumber 0.4160224698E-13 0.1037885340E-08
entropy 0.4527941346E-03 0.7356593433E-11
magvort 0.3309131810E-10 0.4853650509E-08
divu 0.1038209618E-09 0.1547923593E-07
eint_E 50944.00000 0.3010894104E-13
eint_e 50944.00000 0.3010894104E-13
logden 0.3552713679E-13 0.3797364267E-14
StateErr 0.1773834229E-03 0.7819468583E-13
X(C12) 0.1443289932E-14 0.4810966440E-14
X(O16) 0.2553512957E-14 0.3647875652E-14
X(Mg24) 0.2664535259E-14 0.7681683579E-06
abar 0.1598721155E-13 0.1099120792E-14
x_velocity 0.4031177891E-04 0.3858931303E-06
y_velocity 0.3764290415E-04 0.1108030244E-08
z_velocity 0.000000000 0.000000000
t_sound_t_enuc 0.2166523245E-14 0.1852579507E-05
magvel 0.3764290398E-04 0.1108030238E-08
radvel 0.4007988181E-04 0.2033539846E-08
magmom 77934.85141 0.1688501056E-08
angular_momentum_x 0.000000000 0.000000000
angular_momentum_y 0.000000000 0.000000000
angular_momentum_z 0.9472302186E+13 0.1706137433E-08
maggrav 0.000000000 0.000000000
pi 0.3699856627E+15 0.5335643686E-07
pioverp0 0.2993583741E-12 0.3579491289E-07
rhopert 0.1773834229E-03 0.4441469762E-10
tpert 0.8660793304E-02 0.3433251898E-10
I do not think there is any problem here, actually. Even when the EOS is gamma law and all physics (including hydro) is turned off, there's a diff at roundoff level (which is not surprising -- we're still doing something different when there's a ghost zone). If the non-reacting source terms are turned back on, the diffs are larger though still small. But reactions will amplify even roundoff diffs to significantly larger levels.