turbKSP crashes for certain flow conditions

[sseraj]

Description

Using turbKSP ("ANKUseTurbDADI": False) can cause optimizations to crash seemingly randomly with the following error:

[106]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[106]PETSC ERROR: Argument out of range
[106]PETSC ERROR: Inserting a new blocked indexed nonzero block (1231606, 1253914) into matrix
[106]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[106]PETSC ERROR: Petsc Release Version 3.15.1, Jun 17, 2021
[106]PETSC ERROR: #1 MatSetValuesBlocked_MPIBAIJ() at packages/petsc-3.15.1/src/mat/impls/baij/mpi/mpibaij.c:427
[106]PETSC ERROR: #2 MatSetValuesBlocked() at packages/petsc-3.15.1/src/mat/interface/matrix.c:1866
 ---------------------------------------------------------------------------
PETSc or MPI Error. Error Code 63. Detected on Proc **
Error at line:   644 in file: adjointUtils.F90
 ---------------------------------------------------------------------------

The reason for this is not clear, especially because this occurs in the middle of optimizations and analyses in previous iterations work fine.

Code versions

• PETSc: 3.15

[sseraj]

Here is a MWE replicating the issue with just analysis on the MACH-Aero tutorial airfoil mesh. This MWE only works on certain numbers of cores (for example 4, 8, 10). The key to getting this error was using a very small Reynolds number. This makes the turbulence residual large and this is when I suspect turbKSP has problems.

This also explains why we have seen turbKSP crash unpredictably in the middle of optimizations. Some combination of the flow condition and design changes could be unfavorable for the solver.

# Run with mpiexec -np 4 python turbKSP_MWE.py

from adflow import ADFLOW
from baseclasses.problems import AeroProblem

# Set up AeroProblem
ap = AeroProblem(
    name="turbKSP_debug",
    mach=0.3,
    reynolds=10.0,
    reynoldsLength=1.0,
    T=288.15,
    alpha=3.0,
    areaRef=1.0,
    chordRef=1.0,
    evalFuncs=["cl", "cd", "cmy"],
    xRef=0.25,
)

# Set up ADflow
aeroOptions = {
    "gridFile": "MACH-Aero/tutorial/airfoilopt/mesh/n0012.cgns",
    "monitorVariables": ["resrho", "resmom", "resrhoe", "resturb"],
    "ANKUseTurbDADI": False,
}
CFDSolver = ADFLOW(options=aeroOptions)

# Run ADflow
CFDSolver(ap)