hypre
hypre copied to clipboard
hypre-space
From PETSc snes/tutorials/ex56.c ran with hypre, a DIVERGED_INDEFINITE_PC occurs when hypre is configured --with-openmp=1
Hi, I am using hypre via PETSc successfully since years. About 2 weeks ago, I configured PETSc with --with-openmp=1 option activated. It resulted in many failures in our FE code where we are using hypre/BoomerAMG for simple L2 projections.
Digging into the problems, I saw that petsc self tests (ran with make test) seems to fails exactly the same way for some hypre tests, I mean: DIVERGED_INDEFINITE_PC.
It happens for: snes/tutorials/ex56.c
which can be launched with: source /opt/petsc-main/hpclibs.sh cd petsc make snes_tutorials-ex56_hypre
within the petsc docker generated from the last of these 3 dockers files: https://giref.ulaval.ca/~ericc/bug_openmp_petsc_mkl_openmpi/fedora_mkl_and_devtools https://giref.ulaval.ca/~ericc/bug_openmp_petsc_mkl_openmpi/openmpi https://giref.ulaval.ca/~ericc/bug_openmp_petsc_mkl_openmpi/petsc
and built like this:
docker build -t fedora_mkl_and_devtools -f fedora_mkl_and_devtools . docker build -t openmpi -f openmpi . docker build -t petsc -f petsc .
(I am using IntelMKL 2021, OpenMPI 4.1.0 and petsc/main branch here)
When I do:
export OMP_NUM_THREADS=1 make snes_tutorials-ex56_hypre
the test is successful.
Can you help please?
Thanks,
Eric ps: PETSc thread is here: https://lists.mcs.anl.gov/pipermail/petsc-dev/2021-March/027005.html
Hi Eric, I think we need the help of the PETSc folks for this. I have cc’ed Satish. Thanks Ulrike
From: Eric Chamberland @.> Sent: Sunday, March 14, 2021 11:17 AM To: hypre-space/hypre @.> Cc: Subscribed @.***> Subject: [hypre-space/hypre] From PETSc snes/tutorials/ex56.c ran with hypre, a DIVERGED_INDEFINITE_PC occurs when hypre is configured --with-openmp=1 (#303)
Hi, I am using hypre via PETSc successfully since years. About 2 weeks ago, I configured PETSc with --with-openmp=1 option activated. It resulted in many failures in our FE code where we are using hypre/BoomerAMG for simple L2 projections.
Digging into the problems, I saw that petsc self tests (ran with make test) seems to fails exactly the same way for some hypre tests, I mean: DIVERGED_INDEFINITE_PC.
It happens for: snes/tutorials/ex56.c
which can be launched with: source /opt/petsc-main/hpclibs.sh cd petsc make snes_tutorials-ex56_hypre
within the petsc docker generated from the last of these 3 dockers files: https://giref.ulaval.ca/~ericc/bug_openmp_petsc_mkl_openmpi/fedora_mkl_and_devtoolshttps://urldefense.us/v3/__https:/giref.ulaval.ca/*ericc/bug_openmp_petsc_mkl_openmpi/fedora_mkl_and_devtools__;fg!!G2kpM7uM-TzIFchu!mkegDY5FuT4aNkM8FDSYs9kEm-IGmGBFB4Txdlrad-Xo0AMbw0q9fKd0RBo_ismU$ https://giref.ulaval.ca/~ericc/bug_openmp_petsc_mkl_openmpi/openmpihttps://urldefense.us/v3/__https:/giref.ulaval.ca/*ericc/bug_openmp_petsc_mkl_openmpi/openmpi__;fg!!G2kpM7uM-TzIFchu!mkegDY5FuT4aNkM8FDSYs9kEm-IGmGBFB4Txdlrad-Xo0AMbw0q9fKd0RHJC5oCX$ https://giref.ulaval.ca/~ericc/bug_openmp_petsc_mkl_openmpi/petschttps://urldefense.us/v3/__https:/giref.ulaval.ca/*ericc/bug_openmp_petsc_mkl_openmpi/petsc__;fg!!G2kpM7uM-TzIFchu!mkegDY5FuT4aNkM8FDSYs9kEm-IGmGBFB4Txdlrad-Xo0AMbw0q9fKd0RAfT5xBq$
and built like this:
docker build -t fedora_mkl_and_devtools -f fedora_mkl_and_devtools . docker build -t openmpi -f openmpi . docker build -t petsc -f petsc .
(I am using IntelMKL 2021, OpenMPI 4.1.0 and petsc/main branch here)
When I do:
export OMP_NUM_THREADS=1 make snes_tutorials-ex56_hypre
the test is successful.
Can you help please?
Thanks,
Eric ps: PETSc thread is here: https://lists.mcs.anl.gov/pipermail/petsc-dev/2021-March/027005.htmlhttps://urldefense.us/v3/__https:/lists.mcs.anl.gov/pipermail/petsc-dev/2021-March/027005.html__;!!G2kpM7uM-TzIFchu!mkegDY5FuT4aNkM8FDSYs9kEm-IGmGBFB4Txdlrad-Xo0AMbw0q9fKd0RILGuy6Y$
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https:/github.com/hypre-space/hypre/issues/303__;!!G2kpM7uM-TzIFchu!mkegDY5FuT4aNkM8FDSYs9kEm-IGmGBFB4Txdlrad-Xo0AMbw0q9fKd0RCCuK-VI$, or unsubscribehttps://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AD4NLLIIVLX5YTOUJIAJKUDTDT4RDANCNFSM4ZFHIYXQ__;!!G2kpM7uM-TzIFchu!mkegDY5FuT4aNkM8FDSYs9kEm-IGmGBFB4Txdlrad-Xo0AMbw0q9fKd0RE1QyfOs$.
I can reproduce this on my Mac
I don't think the issue is directly related to PETSc and don't see anything more the PETSc team can do in tracking down the problem. Note the problem (if it is in hypre) could likely have been around for many years, we never tested the OpenMP configurations in the past.
It appears when hypre is built with OpenMP support, OMP_NUM_THREADS >= 4 and one uses HYPRE_BoomerAMGSetRelaxOrder with 0 as the relaxation order. The preconditioner becomes indefinite for a positive definite problem indicating a likely problem with the smoother. Note: PETSc does not use OpenMP so will produce the exact same problem for hypre regardless of OMP_NUM_THREADS
git clone [email protected]:petsc/petsc.git
cd petsc
export PETSC_DIR=`pwd`
export PETSC_ARCH=arch-hypre
./configure --with-openmp --download-mpich --download-hypre --download-hypre-commit=origin/master --with-cc=gcc-10 --with-cxx=g++-10 --with-fc=gfortran
make all check
cd src/snes/tutorials
make ex56
export OMP_NUM_THREADS=1
${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -n 2 ./ex56 -cells 2,2,1 -max_conv_its 2 -lx 1. -alpha .01 -petscspace_degree 2 -ksp_type cg -ksp_monitor_short -ksp_rtol 1.e-8 -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_no_CF true -ksp_converged_reason -use_mat_nearnullspace true -petscpartitioner_type simple -ksp_monitor_true_residual
NO ERROR
export OMP_NUM_THREADS=4
${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -n 2 ./ex56 -cells 2,2,1 -max_conv_its 2 -lx 1. -alpha .01 -petscspace_degree 2 -ksp_type cg -ksp_monitor_short -ksp_rtol 1.e-8 -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_no_CF true -ksp_converged_reason -use_mat_nearnullspace true -petscpartitioner_type simple -ksp_monitor_true_residual
0 KSP Residual norm 139.538
0 KSP preconditioned resid norm 1.395378315709e+02 true resid norm 1.223958326481e+02 ||r(i)||/||b|| 1.000000000000e+00
1 KSP Residual norm 7.44006
1 KSP preconditioned resid norm 7.440062105572e+00 true resid norm 2.681088024798e+01 ||r(i)||/||b|| 2.190505972949e-01
2 KSP Residual norm 1.87661
2 KSP preconditioned resid norm 1.876613082946e+00 true resid norm 3.040084975028e+00 ||r(i)||/||b|| 2.483814121162e-02
3 KSP Residual norm 0.163107
3 KSP preconditioned resid norm 1.631071904875e-01 true resid norm 4.009669304336e-01 ||r(i)||/||b|| 3.275985152096e-03
4 KSP Residual norm 0.0222293
4 KSP preconditioned resid norm 2.222929379332e-02 true resid norm 3.742175068092e-02 ||r(i)||/||b|| 3.057436668494e-04
5 KSP Residual norm 0.00196527
5 KSP preconditioned resid norm 1.965271230784e-03 true resid norm 7.250708504822e-03 ||r(i)||/||b|| 5.923983151998e-05
6 KSP Residual norm 0.000386552
6 KSP preconditioned resid norm 3.865524288664e-04 true resid norm 6.760441700802e-04 ||r(i)||/||b|| 5.523424739662e-06
7 KSP Residual norm 2.21708e-05
7 KSP preconditioned resid norm 2.217077975912e-05 true resid norm 3.763036503024e-05 ||r(i)||/||b|| 3.074480904789e-07
8 KSP Residual norm 6.89833e-07
8 KSP preconditioned resid norm 6.898329319343e-07 true resid norm 8.785657136924e-07 ||r(i)||/||b|| 7.178068849927e-09
Linear solve converged due to CONVERGED_RTOL iterations 8
0 KSP Residual norm 429.132
0 KSP preconditioned resid norm 4.291324008768e+02 true resid norm 4.971061216011e+01 ||r(i)||/||b|| 1.000000000000e+00
1 KSP Residual norm 21.1366
1 KSP preconditioned resid norm 2.113661714172e+01 true resid norm 2.687034787505e+01 ||r(i)||/||b|| 5.405354451985e-01
2 KSP Residual norm 6.19201
2 KSP preconditioned resid norm 6.192011137794e+00 true resid norm 2.089686075570e+01 ||r(i)||/||b|| 4.203702156873e-01
3 KSP Residual norm 2.10191
3 KSP preconditioned resid norm 2.101909540691e+00 true resid norm 2.069745808792e+01 ||r(i)||/||b|| 4.163589460789e-01
Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 4
DO NOT USE -pc_hypre_boomeramg_no_CF
${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -n 2 ./ex56 -cells 2,2,1 -max_conv_its 2 -lx 1. -alpha .01 -petscspace_degree 2 -ksp_type cg -ksp_monitor_short -ksp_rtol 1.e-8 -pc_type hypre -pc_hypre_type boomeramg -ksp_converged_reason -use_mat_nearnullspace true -petscpartitioner_type simple -ksp_monitor_true_residual
NO ERROR
Eric, Which smoother are you using? The default or something else? Ulrike
From: Eric Chamberland @.> Sent: Sunday, March 14, 2021 11:17 AM To: hypre-space/hypre @.> Cc: Subscribed @.***> Subject: [hypre-space/hypre] From PETSc snes/tutorials/ex56.c ran with hypre, a DIVERGED_INDEFINITE_PC occurs when hypre is configured --with-openmp=1 (#303)
Hi, I am using hypre via PETSc successfully since years. About 2 weeks ago, I configured PETSc with --with-openmp=1 option activated. It resulted in many failures in our FE code where we are using hypre/BoomerAMG for simple L2 projections.
Digging into the problems, I saw that petsc self tests (ran with make test) seems to fails exactly the same way for some hypre tests, I mean: DIVERGED_INDEFINITE_PC.
It happens for: snes/tutorials/ex56.c
which can be launched with: source /opt/petsc-main/hpclibs.sh cd petsc make snes_tutorials-ex56_hypre
within the petsc docker generated from the last of these 3 dockers files: https://giref.ulaval.ca/~ericc/bug_openmp_petsc_mkl_openmpi/fedora_mkl_and_devtoolshttps://urldefense.us/v3/__https:/giref.ulaval.ca/*ericc/bug_openmp_petsc_mkl_openmpi/fedora_mkl_and_devtools__;fg!!G2kpM7uM-TzIFchu!mkegDY5FuT4aNkM8FDSYs9kEm-IGmGBFB4Txdlrad-Xo0AMbw0q9fKd0RBo_ismU$ https://giref.ulaval.ca/~ericc/bug_openmp_petsc_mkl_openmpi/openmpihttps://urldefense.us/v3/__https:/giref.ulaval.ca/*ericc/bug_openmp_petsc_mkl_openmpi/openmpi__;fg!!G2kpM7uM-TzIFchu!mkegDY5FuT4aNkM8FDSYs9kEm-IGmGBFB4Txdlrad-Xo0AMbw0q9fKd0RHJC5oCX$ https://giref.ulaval.ca/~ericc/bug_openmp_petsc_mkl_openmpi/petschttps://urldefense.us/v3/__https:/giref.ulaval.ca/*ericc/bug_openmp_petsc_mkl_openmpi/petsc__;fg!!G2kpM7uM-TzIFchu!mkegDY5FuT4aNkM8FDSYs9kEm-IGmGBFB4Txdlrad-Xo0AMbw0q9fKd0RAfT5xBq$
and built like this:
docker build -t fedora_mkl_and_devtools -f fedora_mkl_and_devtools . docker build -t openmpi -f openmpi . docker build -t petsc -f petsc .
(I am using IntelMKL 2021, OpenMPI 4.1.0 and petsc/main branch here)
When I do:
export OMP_NUM_THREADS=1 make snes_tutorials-ex56_hypre
the test is successful.
Can you help please?
Thanks,
Eric ps: PETSc thread is here: https://lists.mcs.anl.gov/pipermail/petsc-dev/2021-March/027005.htmlhttps://urldefense.us/v3/__https:/lists.mcs.anl.gov/pipermail/petsc-dev/2021-March/027005.html__;!!G2kpM7uM-TzIFchu!mkegDY5FuT4aNkM8FDSYs9kEm-IGmGBFB4Txdlrad-Xo0AMbw0q9fKd0RILGuy6Y$
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https:/github.com/hypre-space/hypre/issues/303__;!!G2kpM7uM-TzIFchu!mkegDY5FuT4aNkM8FDSYs9kEm-IGmGBFB4Txdlrad-Xo0AMbw0q9fKd0RCCuK-VI$, or unsubscribehttps://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AD4NLLIIVLX5YTOUJIAJKUDTDT4RDANCNFSM4ZFHIYXQ__;!!G2kpM7uM-TzIFchu!mkegDY5FuT4aNkM8FDSYs9kEm-IGmGBFB4Txdlrad-Xo0AMbw0q9fKd0RE1QyfOs$.
Default value set by PETSc (which is not the one of BoomerAMG): relax_type == 6.
Can you change this to relax_type = 8 (which is L1-SGS)? This might help, since hybrid SGS is not guaranteed to converge and affected (more than 8) by OpenMP
From: prj- @.> Sent: Tuesday, March 16, 2021 8:43 AM To: hypre-space/hypre @.> Cc: Yang, Ulrike Meier @.>; Comment @.> Subject: Re: [hypre-space/hypre] From PETSc snes/tutorials/ex56.c ran with hypre, a DIVERGED_INDEFINITE_PC occurs when hypre is configured --with-openmp=1 (#303)
Default value set by PETSc (which is not the one of BoomerAMG): relax_type == 6.
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https:/github.com/hypre-space/hypre/issues/303*issuecomment-800375099__;Iw!!G2kpM7uM-TzIFchu!j_b6f-ljzX4aMchhJgLfKkDgY8aYYBrwVwOGSszwHCZ96xCrZitjsDfgl500vPYx$, or unsubscribehttps://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AD4NLLKYO4FG7MM5SXG52M3TD54AVANCNFSM4ZFHIYXQ__;!!G2kpM7uM-TzIFchu!j_b6f-ljzX4aMchhJgLfKkDgY8aYYBrwVwOGSszwHCZ96xCrZitjsDfgl7NtWFnw$.
We definitely can, but that's not really the point, which is: is it to be expected to get a much worse numerical behavior when using OpenMP?
For that smoother it would be
From: prj- @.> Sent: Tuesday, March 16, 2021 9:15 AM To: hypre-space/hypre @.> Cc: Yang, Ulrike Meier @.>; Comment @.> Subject: Re: [hypre-space/hypre] From PETSc snes/tutorials/ex56.c ran with hypre, a DIVERGED_INDEFINITE_PC occurs when hypre is configured --with-openmp=1 (#303)
We definitely can, but that's not really the point, which is: is it to be expected to get a much worse numerical behavior when using OpenMP?
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https:/github.com/hypre-space/hypre/issues/303*issuecomment-800400795__;Iw!!G2kpM7uM-TzIFchu!gvtgA3k3bxsPXlDvZXwoPOal-a-SE2JPSByQQOepq_91DKFcL6GAS9eHVf8hbxAB$, or unsubscribehttps://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AD4NLLMJIFGGHZFFVPA3WUDTD57XNANCNFSM4ZFHIYXQ__;!!G2kpM7uM-TzIFchu!gvtgA3k3bxsPXlDvZXwoPOal-a-SE2JPSByQQOepq_91DKFcL6GAS9eHVf8iXAUA$.
OK, then I guess @BarrySmith will have to make the executive decision of either turning off --with-openmp in hypre configure or changing the default smoother set by PETSc (maybe only when using OpenMP?).
Yes, generally 8 should be preferred over 6, since it generally guarantees better convergence, but not always. I am just curious if it will work for this problem. Note that it still will depend on the number of threads.
From: prj- @.> Sent: Tuesday, March 16, 2021 9:20 AM To: hypre-space/hypre @.> Cc: Yang, Ulrike Meier @.>; Comment @.> Subject: Re: [hypre-space/hypre] From PETSc snes/tutorials/ex56.c ran with hypre, a DIVERGED_INDEFINITE_PC occurs when hypre is configured --with-openmp=1 (#303)
OK, then I guess @BarrySmithhttps://urldefense.us/v3/__https:/github.com/BarrySmith__;!!G2kpM7uM-TzIFchu!l5jgZGRBVR8PuF0sYllLko37VXJ5bFYk5PcmAN_aqL-Pz5hpT-6wNAozMbZDpPmb$ will have to make the executive decision of either turning off --with-openmp in hypre configure or changing the default smoother set by PETSc (maybe only when using OpenMP?).
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https:/github.com/hypre-space/hypre/issues/303*issuecomment-800405219__;Iw!!G2kpM7uM-TzIFchu!l5jgZGRBVR8PuF0sYllLko37VXJ5bFYk5PcmAN_aqL-Pz5hpT-6wNAozMQ0IiI90$, or unsubscribehttps://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AD4NLLJ6V6DF2L73NOLTEJTTD6AMRANCNFSM4ZFHIYXQ__;!!G2kpM7uM-TzIFchu!l5jgZGRBVR8PuF0sYllLko37VXJ5bFYk5PcmAN_aqL-Pz5hpT-6wNAozMagSwbIK$.
Hi Barry, Can you switch the smoother to L1-SGS (relax_type 8)? That should work better than 6. It still depends on the number of threads, that can’t be avoided, but I hope it will at least converge. Thanks Ulrike
From: Barry Smith @.> Sent: Monday, March 15, 2021 11:42 AM To: hypre-space/hypre @.> Cc: Yang, Ulrike Meier @.>; Comment @.> Subject: Re: [hypre-space/hypre] From PETSc snes/tutorials/ex56.c ran with hypre, a DIVERGED_INDEFINITE_PC occurs when hypre is configured --with-openmp=1 (#303)
I can reproduce this on my Mac
I don't think the issue is directly related to PETSc and don't see anything more the PETSc team can do in tracking down the problem. Note the problem (if it is in hypre) could likely have been around for many years, we never tested the OpenMP configurations in the past.
It appears when hypre is built with OpenMP support, OMP_NUM_THREADS >= 4 and one uses HYPRE_BoomerAMGSetRelaxOrder with 0 as the relaxation order. The preconditioner becomes indefinite for a positive definite problem indicating a likely problem with the smoother. Note: PETSc does not use OpenMP so will produce the exact same problem for hypre regardless of OMP_NUM_THREADS
git clone @.@.:petsc/petsc.git>
cd petsc
export PETSC_DIR=pwd
export PETSC_ARCH=arch-hypre
./configure --with-openmp --download-mpich --download-hypre --download-hypre-commit=origin/master --with-cc=gcc-10 --with-cxx=g++-10 --with-fc=gfortran
make all check
cd src/snes/tutorials
make ex56
export OMP_NUM_THREADS=1
${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -n 2 ./ex56 -cells 2,2,1 -max_conv_its 2 -lx 1. -alpha .01 -petscspace_degree 2 -ksp_type cg -ksp_monitor_short -ksp_rtol 1.e-8 -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_no_CF true -ksp_converged_reason -use_mat_nearnullspace true -petscpartitioner_type simple -ksp_monitor_true_residual
NO ERROR
export OMP_NUM_THREADS=4
${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -n 2 ./ex56 -cells 2,2,1 -max_conv_its 2 -lx 1. -alpha .01 -petscspace_degree 2 -ksp_type cg -ksp_monitor_short -ksp_rtol 1.e-8 -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_no_CF true -ksp_converged_reason -use_mat_nearnullspace true -petscpartitioner_type simple -ksp_monitor_true_residual
0 KSP Residual norm 139.538
0 KSP preconditioned resid norm 1.395378315709e+02 true resid norm 1.223958326481e+02 ||r(i)||/||b|| 1.000000000000e+00
1 KSP Residual norm 7.44006
1 KSP preconditioned resid norm 7.440062105572e+00 true resid norm 2.681088024798e+01 ||r(i)||/||b|| 2.190505972949e-01
2 KSP Residual norm 1.87661
2 KSP preconditioned resid norm 1.876613082946e+00 true resid norm 3.040084975028e+00 ||r(i)||/||b|| 2.483814121162e-02
3 KSP Residual norm 0.163107
3 KSP preconditioned resid norm 1.631071904875e-01 true resid norm 4.009669304336e-01 ||r(i)||/||b|| 3.275985152096e-03
4 KSP Residual norm 0.0222293
4 KSP preconditioned resid norm 2.222929379332e-02 true resid norm 3.742175068092e-02 ||r(i)||/||b|| 3.057436668494e-04
5 KSP Residual norm 0.00196527
5 KSP preconditioned resid norm 1.965271230784e-03 true resid norm 7.250708504822e-03 ||r(i)||/||b|| 5.923983151998e-05
6 KSP Residual norm 0.000386552
6 KSP preconditioned resid norm 3.865524288664e-04 true resid norm 6.760441700802e-04 ||r(i)||/||b|| 5.523424739662e-06
7 KSP Residual norm 2.21708e-05
7 KSP preconditioned resid norm 2.217077975912e-05 true resid norm 3.763036503024e-05 ||r(i)||/||b|| 3.074480904789e-07
8 KSP Residual norm 6.89833e-07
8 KSP preconditioned resid norm 6.898329319343e-07 true resid norm 8.785657136924e-07 ||r(i)||/||b|| 7.178068849927e-09
Linear solve converged due to CONVERGED_RTOL iterations 8
0 KSP Residual norm 429.132
0 KSP preconditioned resid norm 4.291324008768e+02 true resid norm 4.971061216011e+01 ||r(i)||/||b|| 1.000000000000e+00
1 KSP Residual norm 21.1366
1 KSP preconditioned resid norm 2.113661714172e+01 true resid norm 2.687034787505e+01 ||r(i)||/||b|| 5.405354451985e-01
2 KSP Residual norm 6.19201
2 KSP preconditioned resid norm 6.192011137794e+00 true resid norm 2.089686075570e+01 ||r(i)||/||b|| 4.203702156873e-01
3 KSP Residual norm 2.10191
3 KSP preconditioned resid norm 2.101909540691e+00 true resid norm 2.069745808792e+01 ||r(i)||/||b|| 4.163589460789e-01
Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 4
DO NOT USE -pc_hypre_boomeramg_no_CF
${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -n 2 ./ex56 -cells 2,2,1 -max_conv_its 2 -lx 1. -alpha .01 -petscspace_degree 2 -ksp_type cg -ksp_monitor_short -ksp_rtol 1.e-8 -pc_type hypre -pc_hypre_type boomeramg -ksp_converged_reason -use_mat_nearnullspace true -petscpartitioner_type simple -ksp_monitor_true_residual
NO ERROR
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https:/github.com/hypre-space/hypre/issues/303*issuecomment-799659360__;Iw!!G2kpM7uM-TzIFchu!iEBJmaKs6Vj97Pu83h5BX13b6-qrXnaPDH21KAXkClUkEZZuHn-mkzj-Uj2RnkWA$, or unsubscribehttps://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AD4NLLI3MMVLEFBUPS64JDTTDZIHVANCNFSM4ZFHIYXQ__;!!G2kpM7uM-TzIFchu!iEBJmaKs6Vj97Pu83h5BX13b6-qrXnaPDH21KAXkClUkEZZuHn-mkzj-UusHc9ht$.
I have tried:
mpiexec -n 4 ./arch-linux-c-debug/tests/snes/tutorials/ex56 -cells 2,2,1 -max_conv_its 2 -lx 1. -alpha .01 -petscspace_degree 2 -ksp_type cg -ksp_monitor_short -ksp_rtol 1.e-8 -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_no_CF true -pc_hypre_boomeramg_agg_nl 1 -pc_hypre_boomeramg_coarsen_type HMIS -pc_hypre_boomeramg_interp_type ext+i -ksp_converged_reason -use_mat_nearnullspace true -petscpartitioner_type simple -ksp_view
-
adding -pc_hypre_boomeramg_smooth_type schwarz-smoothers it works
-
adding -pc_hypre_boomeramg_smooth_type pilut it works too
-
adding -pc_hypre_boomeramg_smooth_type parasails it also works
-
adding -pc_hypre_boomeramg_smooth_type euclid it works too.
So what sounds strange to me is that if it is known that bad convergence can't be avoided with the first smoother, that is, it is not a bug, then all users should be aware of this? Is it feasible to send some sort of message to the user? Or not allow anyone to use it with over some OMP_NUM_THREADS values? Or is it the burden of all hypre end-users to change their code to not make that "bad" decision? If so, where can we, as end-users developers, found the information written in this issue?
Anyway, thanks for helping!!!
Eric
Eric, that's not quite what Ulrike wanted us to try. Please use instead the option -pc_hypre_boomeramg_relax_type_all l1scaled-SOR/Jacobi (and remove -pc_hypre_boomeramg_smooth_type). Thanks.
Oh, sorry, here it is:
-pc_hypre_boomeramg_relax_type_all l1scaled-SOR/Jacobi
and it works too!
...
0 KSP Residual norm 1.01966e-06
1 KSP Residual norm 3.40277e-07
2 KSP Residual norm 9.85896e-08
3 KSP Residual norm 2.69238e-08
4 KSP Residual norm 7.94589e-09
5 KSP Residual norm 2.11167e-09
6 KSP Residual norm 6.152e-10
7 KSP Residual norm 2.124e-10
8 KSP Residual norm 6.971e-11
9 KSP Residual norm 1.859e-11
10 KSP Residual norm
But my questioning remains...
Eric
I would ask another question:
Are there other configurations that should be carefully chosen when OpenMP is used?
Thanks again, Eric
Ok, that’s good. Actually generally L1-SGS (or l1scaled-SOR/Jacobi as it is apparently called in PETSc) should be used instead of hybrid SGS (not sure what PETSc calls it). Maybe the PETSc default smoother for BoomerAMG should be changed. If someone wants to have a smoother that is not affected by OpenMP, one should use Jacobi or polynomial smoothers, however L1-SGS generally converges better than L1-Jacobi, so I still would recommend this.
From: Eric Chamberland @.> Sent: Tuesday, March 16, 2021 10:48 AM To: hypre-space/hypre @.> Cc: Yang, Ulrike Meier @.>; Comment @.> Subject: Re: [hypre-space/hypre] From PETSc snes/tutorials/ex56.c ran with hypre, a DIVERGED_INDEFINITE_PC occurs when hypre is configured --with-openmp=1 (#303)
Oh, sorry, here it is:
-pc_hypre_boomeramg_relax_type_all l1scaled-SOR/Jacobi
and it works too!
...
0 KSP Residual norm 1.01966e-06
1 KSP Residual norm 3.40277e-07
2 KSP Residual norm 9.85896e-08
3 KSP Residual norm 2.69238e-08
4 KSP Residual norm 7.94589e-09
5 KSP Residual norm 2.11167e-09
6 KSP Residual norm 6.152e-10
7 KSP Residual norm 2.124e-10
8 KSP Residual norm 6.971e-11
9 KSP Residual norm 1.859e-11
10 KSP Residual norm < 1.e-11
11 KSP Residual norm < 1.e-11
12 KSP Residual norm < 1.e-11
13 KSP Residual norm < 1.e-11
14 KSP Residual norm < 1.e-11
15 KSP Residual norm < 1.e-11
16 KSP Residual norm < 1.e-11
Linear solve converged due to CONVERGED_RTOL iterations 16
KSP Object: 4 MPI processes
type: cg
maximum iterations=10000, initial guess is zero
tolerances: relative=1e-08, absolute=1e-50, divergence=10000.
left preconditioning
using PRECONDITIONED norm type for convergence test
PC Object: 4 MPI processes
type: hypre
HYPRE BoomerAMG preconditioning
Cycle type V
Maximum number of levels 25
Maximum number of iterations PER hypre call 1
Convergence tolerance PER hypre call 0.
Threshold for strong coupling 0.25
Interpolation truncation factor 0.
Interpolation: max elements per row 0
Number of levels of aggressive coarsening 1
Number of paths for aggressive coarsening 1
Maximum row sums 0.9
Sweeps down 1
Sweeps up 1
Sweeps on coarse 1
Relax down l1scaled-SOR/Jacobi
Relax up l1scaled-SOR/Jacobi
Relax on coarse Gaussian-elimination
Relax weight (all) 1.
Outer relax weight (all) 1.
Not using CF-relaxation
Not using more complex smoothers.
Measure type local
Coarsen type HMIS
Interpolation type ext+i
linear system matrix = precond matrix:
Mat Object: 4 MPI processes
type: mpiaij
rows=441, cols=441, bs=3
total: nonzeros=33327, allocated nonzeros=33327
total number of mallocs used during MatSetValues calls=0
using I-node (on process 0) routines: found 27 nodes, limit used is 5
[0] 1) N= 441, max displ=5.5560415e+01, disp diff= 1.19e+01, error=4.415e+00, rate=1.9
But my questioning remains...
Eric
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https:/github.com/hypre-space/hypre/issues/303*issuecomment-800476171__;Iw!!G2kpM7uM-TzIFchu!lUPzJ-Tk32XrvWYXFBzw5Wf3PS3GeJdsMpC0A1NjRYTpyJtYb81Eascof1qFZ4AO$, or unsubscribehttps://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AD4NLLJZ6B7IYCSUGEFVLSLTD6KURANCNFSM4ZFHIYXQ__;!!G2kpM7uM-TzIFchu!lUPzJ-Tk32XrvWYXFBzw5Wf3PS3GeJdsMpC0A1NjRYTpyJtYb81Eascof9_MhWNe$.