question about successive preconditioned solves

[bgranzow]

Hello. I have a use case where I would like to perform preconditioned solves over multiple time steps. Initially, I tried an approach where I created both the solver and preconditioner one time up front, called HYPRE_*Solve inside of the time step loop, and then called HYPRE_*Destroy one time at the end of the program. This is illustrated in the code snippet below:

(approach A)

HYPRE_StructSolver solver;
HYPRE_StructSolver precond;
HYPRE_StructPCGCreate(comm, &solver);
HYPRE_StructSMGCreate(comm, &precond);
HYPRE_StructPCGSetPrecond(mg_cg_solver, HYPRE_StructSMGSolve, HYPRE_StructSMGSetup, mg_precond);
for (int step = 0; step < nsteps; ++step) {
  HYPRE_StructPCGSetup(mg_cg_solver, A, b, x);
  HYPRE_StructPCGSolve(mg_cg_solver, A, b, x);
}
HYPRE_StructSMGDestroy(precond);
HYPRE_StructPCGDestroy(solver);

Using this approach, my example program eventually ran out of memory. I switched to creating and destroying the preconditioner every time step (approach B), as illustrated by the code snippet below, and the apparent memory leak was fixed.

(approach B)

HYPRE_StructSolver solver;
HYPRE_StructSolver precond;
HYPRE_StructPCGCreate(comm, &solver);
for (int step = 0; step < nsteps; ++step) {
  HYPRE_StructSMGCreate(comm, &precond);
  HYPRE_StructPCGSetPrecond(mg_cg_solver, HYPRE_StructSMGSolve, HYPRE_StructSMGSetup, mg_precond);
  HYPRE_StructPCGSetup(mg_cg_solver, A, b, x);
  HYPRE_StructPCGSolve(mg_cg_solver, A, b, x);
  HYPRE_StructSMGDestroy(precond);
}
HYPRE_StructPCGDestroy(solver);

I have a few questions:

1. Is approach B the intended usage of HYPRE when performing successive preconditioned solves?
2. If so, are there ways in which approach A could be made to work?

Thanks!

[ibaned]

@liruipeng this is important for our CUDA-capable production code. I think a lot of time is spent just creating the preconditioner every time step, we would love to be able to amortize that.

[ibaned]

Any feedback on this?

[liruipeng]

@ibaned @bgranzow Sorry for replying late. Currently, your approach B is the intended usage, i.e., destroying SMG at each step and creating a new one at the next step, since HYPRE_StructPCGSetup does all the memory allocation so you have to free it. But I don't expect approach A is more efficient since HYPRE_StructSMGCreate and HYPRE_StructPCGSetPrecond should take no time. The cost should be in the setup. If you can reuse SMG preconditioner, it can be amortized but I am not sure if that's possible for your application. Thanks!

[ibaned]

@liruipeng thanks for the reply! Our matrix always has the same structure (non-zero sparsity pattern) and the coefficients change slowly from one time step to the next, so maybe there is a possibility of reusing the preconditioner? Do you have an example of how to do that?

[lahwaacz]

An important difference between the "approach A" and "approach B" discussed above appears when we consider the various settings of the preconditioner. If we need to destroy the preconditioner in each time step after the solve, we also need to set all settings in each time step. If you consider a larger solver where the user can set various solver and preconditioner settings, the "approach B" gets quite annoying as it basically requires duplicating all settings exposed to the user and setting them in the loop instead of letting the preconditioner "remember" the settings across time steps.

Our matrix always has the same structure (non-zero sparsity pattern) and the coefficients change slowly from one time step to the next, so maybe there is a possibility of reusing the preconditioner? Do you have an example of how to do that?

I'm also trying to figure this out. If I understand correctly how Hypre works, the HYPRE_StructPCGSetup function calls the appropriate setup function on the preconditioner, i.e. HYPRE_StructSMGSetup in this case. To reuse the preconditioner, it should be sufficient to avoid calling this setup function in the successive iterations. Can this be done by skipping the whole solver setup or do we need to pass a dummy preconditioner setup function when calling HYPRE_StructPCGSetPrecond?

[lahwaacz]

Also note that the hypre_BoomerAMGSetup function contains the following comment: https://github.com/hypre-space/hypre/blob/07a8def6f8b7dfdffa4208d962f08f5e5d80ccaa/src/parcsr_ls/par_amg_setup.c#L389

So the issue of not freeing memory in a setup function seems to apply only to some preconditioners...

[LiMuxing666]

@ibaned @lahwaacz @bgranzow @jedbrown @liruipeng I've encountered the same issue. Do you have any suggestions? Your help would be greatly appreciated. Thank you.

[LiMuxing666]

It seems I've figured out my issue. I'm using AMS_PCG. After careful observation, I noticed that during the iterative solving process, there are data stored in ams_data. However, it's quite evident that I don't want to destroy the entire ams_data. After some analysis, I realized that the matrix is the part I wish to retain. Then, I need to destroy other data within the AMSsolver, but the most crucial aspect is the BoomerAmg_data within ams_data