Exawind
OpenFAST output frequency is incorrect
Bug description
There's some unexpected behavior in the OpenFAST output file frequency that has crept in recently. Previously, in the OpenFAST fst file, we would have these two parameters:
0.005 DT - Recommended module time step (s)
"default" DT_Out - Time step for tabular output (s) (or "default")
The requirement is that the AMR-Wind dt has to be a fixed multiple of the OpenFAST DT. And by using the "default" for DT_Out, the OpenFAST output file would be written at the OpenFAST dt. This is the expected behavior, and was true when we ran the 16 turbine NREL5MW alm case for HFM in Aug. 2024 (with hash 0230d3ae1cf0243c9a1d06273d179be6700ad810 and openfast 3.5).
However, in more recent versions, I've observed that the OpenFAST output has been set by the AMR-Wind timestep, and is incorrectly set on restarting the case, along with spewing some garbage.
An example, annotated snippet of the incorrect AMR-Wind/OpenFAST output is shown below. In this case, the OpenFAST DT=0.005 s, and the AMR-Wind dt=0.02s. I'm only showing the Time and the Wind1VelX columns for clarity.
899.9400 2.872E+00 # OpenFAST output is at every dt=0.02s (not what I expect)
899.9600 2.871E+00
899.9800 2.871E+00
900.0000 2.870E+00 # Original run stops here
864.0000 3.120E+00 # Restart run starts at t=864 sec
864.00500 4.55179E-01 # Restart output dt=0.005s (at the expected frequency, but the output is garbage)
864.01001 4.55179E-01
864.01501 4.55179E-01
864.02002 4.55179E-01
864.0200 3.120E+00 # there's a duplicated output at dt=0.02s with the correct data
864.02502 4.55179E-01
864.03003 4.55179E-01
864.03497 4.55179E-01
864.03998 4.55179E-01
864.0400 3.121E+00
864.04498 4.55179E-01
864.04999 4.55180E-01
864.05499 4.55180E-01
864.06000 4.55180E-01
864.0600 3.121E+00
On restart, OpenFAST switches to the correct output frequency, but the data is garbage. However, there's duplicated output at the AMR-Wind dt which has the correct information. Also note that the restart OpenFAST output cuts out early, and doesn't last for the entire AMR-Wind run (although OpenFAST is running properly, the output file is just terminated early).
Steps to reproduce
Steps to reproduce the behavior:
-
Compiler used
- [x] GCC
- [ ] LLVM
- [ ] oneapi (Intel)
- [ ] nvcc (NVIDIA)
- [ ] rocm (AMD)
- [ ] with MPI
- [ ] other:
-
Operating system
- [x] Linux
- [ ] OSX
- [ ] Windows
- [ ] other (do tell ;)):
-
Hardware:
- [x] CPU
- [ ] GPU
-
Machine details (): These cases were run on Flight at Sandia, but I'm looking for other examples on Frontier, Kestrel, etc.
-
Input file attachments An example of this can be see in the NREL5MW benchmark problem: https://github.com/Exawind/exawind-benchmarks/blob/main/amr-wind/actuator_line/NREL5MW_ALM_BD/input_files/NREL5MW_ALM_BD_OFv402.inp https://github.com/Exawind/exawind-benchmarks/blob/main/amr-wind/actuator_line/NREL5MW_ALM_BD/setup/nrel5mw_v402_rosco.yaml The turbine model setup instructions are here: https://github.com/Exawind/exawind-benchmarks/blob/main/amr-wind/actuator_line/NREL5MW_ALM_BD/setup/README.md
In this case, OpenFAST is disregarding the explicit DT_Out setting and outputting at the AMR-Wind timestep.
-
Error (paste or attach): No error observed, but the see the OpenFAST output behavior above.
-
If this is a segfault, a stack trace from a debug build (paste or attach): N/A
Expected behavior
See above description for expected behavior.
AMR-Wind information
The problem has probably crept in somewhere over the last year, but this is the most recent AMR-Wind version I've observed it:
==============================================================================
AMR-Wind (https://github.com/exawind/amr-wind)
AMR-Wind version :: v3.6.0-6-g66a70ff8
AMR-Wind Git SHA :: 66a70ff83c1e3f09147c2b5f1a9e32d9450dd5b0
AMReX version :: 25.05-22-gd3798de0bd81
Exec. time :: Sun Sep 21 20:20:40 2025
Build time :: Jul 22 2025 10:33:02
C++ compiler :: GNU 12.1.0
MPI :: ON (Num. ranks = 3584)
GPU :: OFF
OpenMP :: OFF
Enabled third-party libraries:
NetCDF 4.9.2
OpenFAST 4.1
This software is released under the BSD 3-clause license.
See https://github.com/Exawind/amr-wind/blob/development/LICENSE for details.
------------------------------------------------------------------------------
Tagging @ndevelder on this too.
Also, interesting side note, I checked @marchdf's recent 4-turbine NREL5MW blade resolved run, and there the output openFAST is also set at the simulation frequency, not the specified OpenFAST frqeuency. However, in that case, OpenFAST is being driven through Nalu-Wind (although AMR-Wind and Nalu-Wind have the same dt in the hybrid solver). But it's not just a pure AMR-Wind phenomena.
Right now I'm thinking it's a problem to be fixed on both sides.
I believe the reason that DT_Out is being ignored in OpenFAST is in FAST_Subs, where it calculates its own DT_Out if there's a DTdriver being sent in:
IF (PRESENT(DTdriver)) THEN
IF (DTdriver == -1.0_DbKi) THEN
! DTdriver wasn't set, so don't use it
ELSE IF ( ABS( NINT(DTdriver/p%DT) * p%DT - DTdriver ) .lt. 0.001 ) THEN
p%DT_Out = NINT(DTdriver/p%DT) * p%DT
p%n_DT_Out = NINT(DTdriver/p%DT)
ELSE
CALL SetErrStat( ErrID_Fatal, 'DTdriver specified '//TRIM ( Num2LStr( DTdriver ) )//' is not an integral multiple of FAST time step '//TRIM ( Num2LStr( p%DT ) ), ErrStat, ErrMsg, RoutineName )
END IF
END IF
When we initialize through FAST_ExtInfw_Init, it sends in the dtDriver_c being used from AMR-Wind
subroutine FAST_ExtInfw_Init(iTurb_c, TMax, InputFileName_c, TurbIDforName, OutFileRoot_c, &
NumActForcePtsBlade, NumActForcePtsTower, TurbPosn, AbortErrLev_c, &
dtDriver_c, dt_c, InflowType, NumBl_c, NumBlElem_c, NumTwrElem_c, NodeClusterType_c, &
ExtInfw_Input_from_FAST, ExtInfw_Output_to_FAST, ErrStat_c, ErrMsg_c) BIND (C, NAME='FAST_ExtInfw_Init')
However, when we restart and call FAST_ExtInfw_Restart, I think dtDriver doesn't get passed and OpenFAST has no knowledge of it.
subroutine FAST_ExtInfw_Restart(iTurb_c, CheckpointRootName_c, AbortErrLev_c, dt_c, numblades_c, numElementsPerBlade_c, numElementsTower_c, n_t_global_c, &
ExtInfw_Input_from_FAST, ExtInfw_Output_to_FAST, ErrStat_c, ErrMsg_c) BIND (C, NAME='FAST_ExtInfw_Restart')
So now OpenFAST thinks it has to output every timestep. However, the OpenFAST marching is still synced up with AMR-Wind.
That's my hypothesis so far anyway. Might have to get some confirmation from the OpenFAST team (@deslaughter?) and see how we want to fix it.
Lawrence
And just to be totally clear, I think maybe we want all the following to be true:
- OpenFAST controls its' own output dt....so that anything you set in the OpenFAST file gets honored. If the cfd solver controls this, you could never get OpenFAST output that is at a higher sample rate than the cfd timestep
- CFD solver triggers checkpoints in OpenFAST on the restart frequency...which is current behavior and slightly unrelated? There are a couple places in the input file this exists, and the behavior could be documented better I think?
- If OpenFAST output is at outlandish frequency, maybe we give a warning?
What do people think about this? Anything to add?
Although this is something that might have to be addressed in a different issue, I agree with this:
CFD solver triggers checkpoints in OpenFAST on the restart frequency...which is current behavior and slightly unrelated? There are a couple places in the input file this exists, and the behavior could be documented better I think?
I also wanted to add that currently AMR-Wind will write a checkpoint at the end of the run, even if it's not at checkpoint_interval. However, it doesn't trigger OpenFAST to write a chkp checkpoint, so users can't pick up right where a simulation leaves off.
Lawrence
This issue is stale because it has been open 30 days with no activity.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}