scons Fortran compile fails with module dependencies

Describe the bug scons does not appear to be correctly find dependencies on *.mod files in Fortran files.

My project includes a Fortran library that includes a number of source files and a module including some shared variables. When compiling on windows with ifort, I get the following error when doing a clean build: error #7002: Error in opening the compiled module file. Check INCLUDE paths. Subsequent builds complete successfully.

Investigation shows that the above compiler error is seen because the *.mod file required by the source files is not built before the source file compilation. Reordering the list of source files improves the situation, but it still fails periodically when building in parallel. Subsequent attempts to compile work as the *.mod file is created during the first scons attempt and are thus already available.

Per suggestion from discord, I ran scons with --tree=prune to check if the consumers of the module correctly listed the *.mod file as a build dependency. The output below (anonymized) shows the output for one of the source files:

  |   +-<path>\<objName>.staticrt.obj
  |   | +-<relativePath>\<srcFile>.f
  |   | +-C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.5.281\windows\bin\intel64\ifort.EXE

Note that the source file does not list the *.mod file as a dependency.

The project is cross-platform so I also checked the result of --tree=prune on linux (compiled with gfortran). Scons also did not recognize the dependency on the mod file (output was the same as above with the exception of the compiler path). However, the code compiles without issue on linux.

Since the above shows that the source files are not identifying the *.mod file as a build dependency, my assumption is that there is some issue in the way that scons is determining dependencies for Fortran files. I'm not sure why there's not an issue with gfortran on linux.

Required information

Link to SCons Users thread discussing your issue. discussed on the scons discord (fortran-support channel)
Version of SCons 3.1.2
Version of Python 3.7.13
Which python distribution if applicable (python.org, cygwin, anaconda, macports, brew,etc) cpython
How you installed SCons pip install
What Platform are you on? (Linux/Windows and which version) Windows 10 Enterprise version1909
How to reproduce your issue? Please include a small self contained reproducer. Likely a SConstruct should do for most issues.
How you invoke scons (The command line you're using "scons --flags some_arguments") scons -j8

Jun 15 '22 15:06 james-thunes

I haven't been able to get a small test case set up yet. I will try to create one and post to this issue

Jun 15 '22 16:06 james-thunes

I'll just add that when I was testing this locally it seems to only be reproducible when running a parallel build with -jN. A serial build would do things in the right order and I never saw that fail, but for parallel it's failing when a dependent source file is compiled concurrently before the module is complete.

Jun 15 '22 16:06 dnwillia-work

I'll just add that when I was testing this locally it seems to only be reproducible when running a parallel build with -jN. A serial build would do things in the right order and I never saw that fail, but for parallel it's failing when a dependent source file is compiled concurrently before the module is complete.

That's just luck... If parallel builds fail due to missing dependencies (for any build system), it means the build system doesn't have all the appropriate dependencies and so isn't building everything needed for the failing build step. I run into this often (though mainly with non-scons based build systems where it's harder to get this right)..

Jun 15 '22 16:06 bdbaddog

Yeah, fair enough. Even for the parallel builds I'll add that it would sometimes work, sometimes not.

I wonder since it seems to be a windows specific thing then is something else going on. eg: the .mod file is still locked by the OS so the dependent module cannot link against it. We have run into this kind of issue with .lib (import library) files on windows parallel builds before as well if an Install() was used to move the .lib file to some other place before it was linked into dependent module despite the fact that we would add Depends() on the install task to the dependent module. We removed the use of Install() targets for this as a result and it got better.

Jun 15 '22 16:06 dnwillia-work

Ugh.. the joys of NTFS..

Jun 15 '22 16:06 bdbaddog

How are you creating your Environment()? Specifically are you setting FORTRANSUFFIXES (or any of the F*SUFFIXES env vars)?

Jun 15 '22 17:06 bdbaddog

Try adding this to your Environment()

FORTRANFILESUFFIXES=['.f90'],

Jun 15 '22 17:06 bdbaddog

this particular library has all *.f files, but I can try setting FORTRANFILESUFFIXES=['.f'],

Jun 15 '22 17:06 james-thunes

How are you creating your Environment()? Specifically are you setting FORTRANSUFFIXES (or any of the F*SUFFIXES env vars)?

We definitely do not configure that particular variable currently. We do configure these variables related to this issue

    env["FORTRANMODDIR"] = "${TARGET.dir}"
    env["F90PATH"] = "${TARGET.dir}"

It's the same for ifort and gfortran.

Jun 15 '22 17:06 dnwillia-work

Okay, I've located a simple example and don't see the problem with it. Here's the tree dump:

+-.
  +-SConstruct
  +-list-module.f90
  +-list-module.o
  | +-list-module.f90
  | +-/bin/gfortran
  +-listmodule.mod
  | +-list-module.f90
  | +-/bin/gfortran
  +-main-list
  | +-main-list.o
  | | +-main-list.f90
  | | +-[listmodule.mod]
  | | +-/bin/gfortran
  | +-[list-module.o]
  | +-/bin/gfortran
  +-main-list.f90
  +-[main-list.o]

main-list.o is shown as depending on its source file, on listmodule.mod, and on the compiler. That looks right to me? What am I missing?

Jun 17 '22 16:06 mwichmann

@mwichmann - I think this is only an intel fortran issue. And I think it's due to loading default tools, then ifort and having files named .f when ifort looks like it only wants files named .i or .i90?

Jun 17 '22 16:06 bdbaddog

this particular library has all *.f files, but I can try setting FORTRANFILESUFFIXES=['.f'],

Did that work?

Jun 17 '22 16:06 bdbaddog

I still have to look at the .i/.i90 thing, I think it should work just find on ordinary suffixes and I don't know what those actually mean.

Jun 17 '22 16:06 mwichmann

this particular library has all *.f files, but I can try setting FORTRANFILESUFFIXES=['.f'],

Did that work?

Sorry, been busy with other tasks. Will try to get to this as soon as I can.

Jun 17 '22 16:06 james-thunes

this particular library has all *.f files, but I can try setting FORTRANFILESUFFIXES=['.f'],

Did that work?

Ok, I was able to spend a bit of time with this today. I'm a bit confused to be honest. Setting FORTRANFILESUFFIXES=['.f'], did indeed appear to resolve the issue. I was able to see the module dependencies when looking at the output with --tree=prune. However, removing the flag and doing a clean rebuild of the code I see that the module dependencies is still seen. It definitely wasn't last week...

Jun 20 '22 21:06 james-thunes

Most of this setup bemuses me, so you have company :)

Out of curiosity, are all the files in your project of the same suffix?

Jun 20 '22 21:06 mwichmann

The library in question has a combination of both .f90 and .f suffixes since it includes F90 free format and F77 fixed format code.

Jun 20 '22 22:06 dnwillia-work

Okay, then the muck with "dialects" isn't completely useless to you (or if it is - please feel free to let us know!).

Jun 20 '22 22:06 mwichmann

Right, the library in question uses both dialects because it links in an older code base shared with other applications. So, definitely useful to us. One issue though is that it's still valid to have F90/95 etc... code in fixed form files with a .f extension, so it muddies the water some more.

I tried the --tree=prune thing with the latest SCons. SCons by Steven Knight et al.: SCons: v4.3.0.559790274f66fa55251f5754de34820a29c7327a, Tue, 16 Nov 2021 19:09:21 +0000, by bdeegan on octodog

on the problematic code and I get the following

  | +-<path-to-lib>\importLibrary.lib
  |   +-<path-obj>\module1.obj
  |   | +-<path-src>\module1.f
  |   | +-C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.5.281\windows\bin\intel64\ifort.EXE
  |   +-<path-obj>\module2.obj
  |   | +-<path-src>\module2.f
  |   | +-C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.5.281\windows\bin\intel64\ifort.EXE
  |   +-<path-obj>\fortranFile1.obj
  |   | +-<path-src>\fortranFile1.f
  |   | +-C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.5.281\windows\bin\intel64\ifort.EXE

In this instance, fortranFile1.f has both use module1 and use module2 statements within the contained functions/subroutines, yet the dependency is not showing up in the tree. However, if I look further down in the tree I see this:

  |   +-<path-obj>\fortranFile2.obj
  |   | +-<path-src>\fortranFile2.f90
  |   | +-<path-obj>\moduleA.mod
  |   | | +-<path-src>\moduleA.f90
  |   | | +-<path-obj>\moduleB.mod
  |   | | | +-<path-src>\moduleB.f90
  |   | | | +-C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.5.281\windows\bin\intel64\ifort.EXE
  |   | | +-C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.5.281\windows\bin\intel64\ifort.EXE
  |   | +-[<path-obj>\moduleB.mod]
  |   | +-C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.5.281\windows\bin\intel64\ifort.EXE

So, it seems to be getting the dependency correct for .f90 extensions but not .f extensions.

Could it be that the .mod files are not added here properly when we are passing .f files that use modules to the library target?

Jun 21 '22 17:06 dnwillia-work

So, it seems to be getting the dependency correct for .f90 extensions but not .f extensions.

That's not impossible, since they use separate settings groups, but this stuff is convoluted enough it's hard to see. There are also slightly differing settings for lib and non-lib.

Did you say you were using a modified copy of ifort.py?

Is it possible to attach a dump of the environment variables (really only the set that start with F are interesting, I believe) - it might show the anomaly that's hitting your usage.

Jun 21 '22 17:06 mwichmann

Yes, we have a modified version of ifort.py. Staring at the differences with your current code the diffs are:

It configures msvc tool if needed. This could maybe be removed.
We have a few functions that hunt for the installed location of the compiler, gets all the versions etc... and then picks the latest. I think I got this code from your project at some point, it seems to have been disappeared.
Sets the variables FORTRANMODDIR and F90PATH as I mentioned earlier

Here's the variables:

FORTRANFLAGS = ['/nologo', '/MD', '/Qvc14', '/warn:unused', '/warn:uncalled', '/O3'] # We set this one
F90FLAGS = ['/nologo', '/MD', '/Qvc14', '/warn:unused', '/warn:uncalled', '/O3'] # We set this one
F95FLAGS = ['/nologo', '/MD', '/Qvc14', '/warn:unused', '/warn:uncalled', '/O3'] # We set this one
FORTRANPATH = ['#.']
FORTRANSUFFIXES = ['.f', '.for', '.ftn', '.F', '.FOR', '.FTN', '.fpp', '.FPP', '.f77', '.F77', '.f90', '.F90', '.f95', '.F95', '.f03', '.F03', '.f08', '.F08']
FORTRANCOM = $FORTRAN -object:$TARGET -c $FORTRANFLAGS $_FORTRANINCFLAGS $_FORTRANMODFLAG $SOURCES
FORTRANPPCOM = $FORTRAN -object:$TARGET -c $FORTRANFLAGS $CPPFLAGS $_CPPDEFFLAGS $_FORTRANINCFLAGS $_FORTRANMODFLAG $SOURCES
FORTRANMODPREFIX =
FORTRANMODSUFFIX = .mod
FORTRANMODDIR = ${TARGET.dir} # We set this one
FORTRANMODDIRPREFIX = /module:
FORTRANMODDIRSUFFIX =
F77FLAGS =
F77COM = $F77 -object:$TARGET -c $F77FLAGS $_F77INCFLAGS $SOURCES
F77PPCOM = $F77 -object:$TARGET -c $F77FLAGS $CPPFLAGS $_CPPDEFFLAGS $_F77INCFLAGS $SOURCES
F90COM = $F90 -object:$TARGET -c $F90FLAGS $_F90INCFLAGS $_FORTRANMODFLAG $SOURCES
F90PPCOM = $F90 -object:$TARGET -c $F90FLAGS $CPPFLAGS $_CPPDEFFLAGS $_F90INCFLAGS $_FORTRANMODFLAG $SOURCES
F95COM = $F95 -object:$TARGET -c $F95FLAGS $_F95INCFLAGS $_FORTRANMODFLAG $SOURCES
F95PPCOM = $F95 -object:$TARGET -c $F95FLAGS $CPPFLAGS $_CPPDEFFLAGS $_F95INCFLAGS $_FORTRANMODFLAG $SOURCES
F03FLAGS =
F03COM = $F03 -o $TARGET -c $F03FLAGS $_F03INCFLAGS $_FORTRANMODFLAG $SOURCES
F03PPCOM = $F03 -o $TARGET -c $F03FLAGS $CPPFLAGS $_CPPDEFFLAGS $_F03INCFLAGS $_FORTRANMODFLAG $SOURCES
F08FLAGS =
F08COM = $F08 -o $TARGET -c $F08FLAGS $_F08INCFLAGS $_FORTRANMODFLAG $SOURCES
F08PPCOM = $F08 -o $TARGET -c $F08FLAGS $CPPFLAGS $_CPPDEFFLAGS $_F08INCFLAGS $_FORTRANMODFLAG $SOURCES
F77 = ifort
F90 = ifort
FORTRAN = ifort
F95 = ifort
F90PATH = ${TARGET.dir}  # We set this one

Noted which ones we explicitly set.

Jun 21 '22 19:06 dnwillia-work

Based on my hypothesis I tried to reproduce this with a simple example, could not do it yet....

Jun 21 '22 20:06 dnwillia-work

@dnwillia-work - could you provide your modified version of ifort.py ?

Jun 22 '22 01:06 bdbaddog

Yeah sure. I’ll push it up later today and share a PR.

Jun 22 '22 10:06 dnwillia-work

At least at my end, the effort of the ifort.py tool to add those extra recognized suffixes (.i and .i90) messes things up - you get into a place where apparently the .f files aren't scanned (the .f90 ones aren't either, so this doesn't explain your issue). Are those actually used for anything? As I said... somewhere... I can't find any hint of those in current Intel docs, which aren't particularly searchable.

Jun 22 '22 15:06 mwichmann

Regarding the file extensions there is this:

https://www.intel.com/content/www/us/en/develop/documentation/fortran-compiler-oneapi-dev-guide-and-reference/top/compiler-setup/use-the-command-line/file-extensions.html

So, Intel does seem to accept .i and .i90 as acceptable file extensions where those files are passed to the compiler.

Jun 22 '22 15:06 dnwillia-work

Sigh... I'd swear I had been looking at that very page, must be blind.

Jun 22 '22 15:06 mwichmann

lol. Yeah, it's not the easiest to navigate or digest it.

Here's the ifort tool module we are using currently:

https://github.com/dnwillia-work/scons/pull/1

Jun 22 '22 16:06 dnwillia-work

There are these comments in the wiki

Jun 22 '22 17:06 mwichmann

Thanks for sharing that. In general, I agree with what is proposed. The stuff about the dialects is probably overly complex.

I tried again to reproduce this issue outside our regular build system using the ifort.py tool I shared and cannot. My SConstruct looks like this:

import os
localTools = os.path.join('.', 'site_scons', 'site_tools')
env = Environment(toolPath = localTools, tools = ['default', 'ifort'])
env['LINKFLAGS'] += ['/subsystem:console'] # Needed to link a Program()
env.Program('hello77', ['hello77.f', 'fixedModule.f', 'freeModule.f90'])
env.Program('hello90', ['hello90.f90', 'fixedModule.f', 'freeModule.f90'])

Source files are:

hello77.f

      program hello
      use fixedModule, only : sayHello
      use freeModule, only : sayHello90
      call sayHello
      call sayHello90
      end program hello

hello90.f90

program hello90
  use fixedModule, only : sayHello
  use freeModule, only : sayHello90
  call sayHello
  call sayHello90
end program hello90

freeModule.f90

module freeModule
  public :: sayHello90
contains
  subroutine sayHello90()
    print *,"Hello from free form module!"
  end subroutine sayHello90
end module freeModule

fixedModule.f

      module fixedModule
      public :: sayHello
      contains
      subroutine sayHello()
        print *,"Hello from fixed form module!"
      end subroutine sayHello
      end module fixedModule

I get this tree:

+-.
  +-fixedModule.f
  +-fixedmodule.mod
  | +-fixedModule.f
  | +-C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.5.281\windows\bin\intel64\ifort.EXE
  +-fixedModule.obj
  | +-fixedModule.f
  | +-C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.5.281\windows\bin\intel64\ifort.EXE
  +-freeModule.f90
  +-freemodule.mod
  | +-freeModule.f90
  | +-C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.5.281\windows\bin\intel64\ifort.EXE
  +-freeModule.obj
  | +-freeModule.f90
  | +-C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.5.281\windows\bin\intel64\ifort.EXE
  +-hello77.exe
  | +-hello77.obj
  | | +-hello77.f
  | | +-[fixedmodule.mod]
  | | +-[freemodule.mod]
  | | +-C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.5.281\windows\bin\intel64\ifort.EXE
  | +-[fixedModule.obj]
  | +-[freeModule.obj]
  | +-C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.20.27508\bin\HostX64\x64\link.EXE
  +-hello77.f
  +-[hello77.obj]
  +-hello90.exe
  | +-hello90.obj
  | | +-hello90.f90
  | | +-[fixedmodule.mod]
  | | +-[freemodule.mod]
  | | +-C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.5.281\windows\bin\intel64\ifort.EXE
  | +-[fixedModule.obj]
  | +-[freeModule.obj]
  | +-C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.20.27508\bin\HostX64\x64\link.EXE
  +-hello90.f90
  +-[hello90.obj]
  +-SConstruct

So, it all looks ok.

Interesting side point: I have to add /subsystem:console there for env.Program() targets to link otherwise the linker does not find the entry point.

Jun 22 '22 18:06 dnwillia-work