Stack overflow in nf90_get_var routines (or something else?)

[arthurvd]

Hi,

We're running into stack overflow issues when calling nf90_get_var routines such as:

function nf90_get_var_2D_FourByteInt(ncid, varid, values, start, count, stride, map)
  integer,                         intent( in) :: ncid, varid
  integer (kind = FourByteInt), dimension(:, :), &
                                   intent(out) :: values
! [..]
  integer, dimension(size(values))      :: defaultIntArray

(from file fortran/netcdf_expanded.f90.) The variable defaultIntArray gives the local stack overflow. The file that I'm reading is a NetCDF classic file, size approx 47 MB and the variable read that crashes has dimensions (2, 387548). (But the same stack overflow occurs on rank-1 arrays.)

My question is: is this still the appropriate code (I see the latest changes from 6 years ago)? And if so: how to fix it? The code can of course be changed using allocatables, but have others maybe circumvented this problem with bigger stack sizes or different compiler flags?

Our setup:

• code from v.4.4.4 Released May 13, 2016
• compiled on Windows using Visual Studio 2012+ Intel(R) Visual Fortran Composer XE 2013 SP1 Update 3 (14.0.0092.11)

[arthurvd]

@edhartnett @WardF Would be great if you could shed light on part of the above questions. Specifically: the above code from fortran/netcdf_expanded.f90 that my call ends up in: are those the correct interfaces? (given that the files have their last change from six years ago). I'm a bit confused that we would be the first ones to run into this issue (since the cause of the stack overflow is a very straightforward programming mistake in fortran).

[WardF]

Perhaps this is something that is handled by optimization at the compiler level on most platforms; Visual Studio 2012 isn't an officially supported compiler. I'll take a look at this and see what we can do on our end, but perhaps compiling with a bigger stack size or exploring the compiler flags on your end may be helpful in the short term.

[arjenmarkus]

I discussed this with Arthur (we are colleagues) and I came up with a small test program:

`program ncproblem use netcdf

implicit none

!real(kind=kind(1.0d0)), dimension(2,4000000) :: data
!integer, dimension(2,387548)                 :: data
integer, dimension(2,400000)                 :: data
integer                                      :: error
integer                                      :: ncid, varid
integer                                      :: dim1, dim2

data  = 1

error = nf90_create( "ncproblem.nc", ior(nf90_clobber, nf90_netcdf4), ncid )

error = nf90_def_dim( ncid, "First",  size(data,1), dim1 )
error = nf90_def_dim( ncid, "Second", size(data,2), dim2 )

error = nf90_def_var( ncid, "MyData", nf90_int, [dim1, dim2], varid )

error = nf90_enddef( ncid )

error = nf90_put_var( ncid, varid, data )
write(*,*) 'Put var: ', error

data  = 0.0
error = nf90_get_var( ncid, varid, data )
write(*,*) 'Get var: ', error
write(*,*) 'Data:    ', sum(data)

error = nf90_close( ncid )

end program ncproblem ` When I use the real version (either single or double precision), the program succeeds splendidly (using all default compile options). However, the version I just posted crashes:

`...>ncproblem Put var: 0 forrtl: severe (170): Program Exception - stack overflow Image PC Routine Line Source

ncproblem.exe 000000013FB8F5C8 Unknown Unknown Unknown ncproblem.exe 000000013FAE1EE2 Unknown Unknown Unknown ncproblem.exe 000000013FAB1378 Unknown Unknown Unknown ncproblem.exe 000000013FB8DF8E Unknown Unknown Unknown ncproblem.exe 000000013FB8F505 Unknown Unknown Unknown kernel32.dll 00000000777159CD Unknown Unknown Unknown ntdll.dll 000000007787383D Unknown Unknown Unknown `

I will try and see whether stack size options can cure this.

[arjenmarkus]

Hm, if I set the stack size to 10 MB (/F10000000), then the program succeeds. /F is a linker option.

An alternative might be the option /heap-arrays:n (n minimum size in kB). That does not work on the program itself, as it is a compile option, so I would have to specify it when compiling the NetCDF Fortran interface itself. That is certainly a possibility worth pursuing, as then the burden of specifying the option does not lie with all programs that use the library.

[arjenmarkus]

Just tried it with that option (rebuilding the NetCDF Fortran interface with /heap-arrays:100) and that did do the trick! So I can build the test program without any special options now :).