Allow `library.source-dir` have multiple sources

[14NGiestas]

Description

I have this legacy project that had the brilliant idea (/s) of selecting functionality based on including only certain directories during the compilation process. In simpler terms, it is something like this:

src/
   main.f90
   model_X/
      specific_file.f90
   model_Y/
      specific_file.f90
   ...
   param_X/
      specific_file.f90
   param_Y/
      specific_file.f90

If I want model_X with param_Y to be used I compile the code setting the respective variables [yeah as if it was a poor's man config file] in the Makefile and hope for the best.

The current build process is messy and I want to simplify it using fpm, as the include-dir key already supports multiple folder names to include files from I wonder if it is possible to source-dir also support that.

[ library ]
source-dir = ["src/main/", "src/model_X", "src/param_Y"]

Possible Solution

With the current fpm capabilities, I could:

• rewrite the code from scratch (I already did that :wink: but I think maybe it is a good thing to have the old one around for comparisons).
• create a local package for each and add/remove it as a dependency (that will require 13 fpm packages)
• manually move the folders around every time or use system links
• cease and desist and try to fix 1530 lines of makefiles

While with that feature I could alter the name of the files and it would be easier to port them to fpm.

Additional Information

While my use case is hacky[1] I suppose there may be other old projects that could benefit from such a feature, plus it's backward compatible as far as I can see, as it would still allow just using it string for a single source.

relevant code 1 relevant code 2 relevant code 3

Also, it looks fairly simple to implement so I could write a PR myself to see if this idea really works in practice.

[1] that is not even the weirdest thing about it as it uses preprocessing to support numeric portability and select other functionality as well (why not?);

[urbanjost]

Something that fpm(1) can accommodate now that I think fits your description would be ...

Since you are already using a preprocessor, the easiest way is probably to put your difference versions into the include/ directory, and then use preprocessor directives in the files in the src/ directory. So for example you could build the different combinations with

fpm run --flag '-DX1 -DY1'
fpm run --flag '-DX2 -DY1'
fpm run --flag '-DX1 -DY2'
fpm run --flag '-DX2 -DY2'

Note that the files in src/ contain the routine or module names so fpm(1) finds the proper names; which with modules is simple but is more cumbersome with files just containing procedures.

So in this case the file layout is

├── app
│   └── main.f90
├── fpm.toml
├── include
│   ├── M_X_1.f90
│   ├── M_X_2.f90
│   ├── M_Y_1.f90
│   └── M_Y_2.f90
├── README.md
├── src
│   ├── M_X.F90
│   └── M_Y.F90

app/main.f90:

program main
   Use M_X, only : helloX=>say_hello
   Use M_Y, only : helloY=>say_hello
   implicit none
     print *, "hello from project top"
     call helloX()
     call helloY()
   end program main

include/M_X_1.f90:

  implicit none
  private
  public :: say_hello
  contains
  subroutine say_hello
    print *, "Hello, X1!"
  end subroutine say_hello

include/M_X_2.f90:

   implicit none
   private
   public :: say_hello
   contains
   subroutine say_hello
      print *, "Hello, X2!"
   end subroutine say_hello

include/M_Y_1.f90:

     implicit none
     private
     public :: say_hello
   contains
     subroutine say_hello
       print *, "Hello, Y1!"
     end subroutine say_hello

include/M_Y_2.f90:

     implicit none
     private
     public :: say_hello
   contains
     subroutine say_hello
       print *, "Hello, Y2!"
     end subroutine say_hello

src/M_X.F90:

   module M_X
   #if X1
   #include "M_X_1.f90"
   #endif
   #if X2
   #include "M_X_2.f90"
   #endif
   end module M_X

src/M_Y.F90:

   module M_Y
   #if Y1
   #include "M_Y_1.f90"
   #endif
   #if Y2
   #include "M_Y_2.f90"
   #endif
   end module M_Y

You can also do this by making the subdirectories you currently have into fpm packages, which is probably quite simple. Go into your current directories and make a subdirectory called src/ and move your Fortran files into that, and then enter "fpm new . --backfill". and then change your fpm.toml file to look something like:

name = "top"
version = "0.1.0"
license = "license"
author = "John S. Urban"
maintainer = "[email protected]"
copyright = "Copyright 2022, John S. Urban"
[build]
auto-executables = false
[install]
library = false

[[executable]]
name = "x1y1"
source-dir = "app"
main = "main.f90"
[executable.dependencies]
X1 = { path = "X1" }
Y1 = { path = "Y1" }

[[executable]]
name = "x1y2"
source-dir = "app"
main = "main.f90"
[executable.dependencies]
X1 = { path = "X1" }
Y2 = { path = "Y2" }

[[executable]]
name = "x2y2"
source-dir = "app"
main = "main.f90"
[executable.dependencies]
X2 = { path = "X2" }
Y2 = { path = "Y2" }

[[executable]]
name = "x2y1"
source-dir = "app"
main = "main.f90"
[executable.dependencies]
X2 = { path = "X2" }
Y1 = { path = "Y1" }

where X1 and X2 are your directory names. This works very nicely and does not require --flag options but because fpm(1) thinks it is seeing duplicate modules you have to comment out all but one executable and run "fpm clean --all" when you switch to a different build. I have a similar situation where I have a bash script that creates a fpm.toml with the paths I want to use which makes that easy, which is called "fpm-select" that lets me just enter "fpm select X1 Y2" and then use fpm with no additional flags until I want to change to another model, then I just enter "fpm select X2 Y1". That works nicely except for having to clean and rebuild between model changes. If fpm(1) did not see the modules as duplicates even though they are only used as dependencies at the executable level, not at the package level, you could just enter "fpm build" and have all four of the combinations shown build, which would be nicer.

It would depend exactly on how many files and combinations you have, and whether they are modules and so on as to which method might work best. Is your code currently on a public github/gitlab/... site?

[14NGiestas]

Thank you for your careful response,

I guess I'll settle on your suggestion of using local packages for now, it's the cleaner solution and the easiest to achieve.

Since you are already using a preprocessor

As I wish I could remove all the current preprocessing, I don't want to introduce any more of it.

It would depend exactly on how many files and combinations you have, and whether they are modules and so on as to which method might work best.

There are 9 models, and each consists of about 10 files and 4 parameters with 2 files.

So there are about 98 files, and theoretically, that would give 36 combinations of model and parameter (I don't know for sure if every comb. is valid). However, I don't need to explicitly type every one of them I'll just let that in the user's hand.

"fpm new . --backfill".

That works (with a --lib is even better), I'll just need to move the files to src for each folder which is just a little boring but the end result is worth it.

Is your code currently on a public github/gitlab/... site?

Unfortunately no, the legacy one is not mine and it isn't public (I guess for good xD);

[14NGiestas]

There is yet another issue with local dependencies... fpm new will create a git inside a git and git warns me that. So I'll still need to remove the .git/

warning: adding embedded git repository: <path>
hint: You've added another git repository inside your current repository.
hint: Clones of the outer repository will not contain the contents of
hint: the embedded repository and will not know how to obtain it.
hint: If you meant to add a submodule, use:
hint: 
hint: 	git submodule add <url> 
hint: 
hint: If you added this path by mistake, you can remove it from the
hint: index with:
hint: 
hint: 	git rm --cached <path>
hint: 
hint: See "git help submodule" for more information.

The end command I'm using is:

fpm new . --backfill --lib; rm -rf .git; mv *.f90 src

[urbanjost]

leaving each subdirectory as totally independent fpm/git projects works so you can ignore the warning and maintain the directories as projects; but it means if you are using git(1) you have to make commits individually in each one and that the top directory will ignore the subdirectories. So it is not necessary in general to remove the directories; unless you want to maintain everything together in one git repository (which it sounds like you probably do want to, unless those subdirectories are also used by other projects, in which case dividing it up into separate fpm projects works well in my experience).

I just wanted to describe a few alternate ways to use fpm(1) that I think are not all that obvious; the use of local dependencies and the include/ directory are not heavily emphasized in the documentation and allow for a little more flexibility than the basic layout. Some additional features like you describe might still be desirable; but keeping the layout fpm(1) uses as standard as possible does make it easier to navigate through others' packages when using a lot of dependencies and looking through them (although the goal is to use external dependencies pretty much as a "black box", I suppose).

Doing basically the same thing but leaving the fpm.toml file alone but having it point to paths that are links and then making a simple script to change where the links point, which I think you alluded to in your list of possible solutions seems like it would be good as well,

[14NGiestas]

but it means if you are using git(1) you have to make commits individually in each one

Yeah, since I need to move the content to src after the fpm new changes are tracked in the newly created git repo, that would require me to also commit in each new package otherwise I couldn't commit the top-level repo... just removing the .git did the trick xD.

Such a use case may be worth a tutorial at fpm-docs if I successfully port the code I may try writing some.

Some additional features like you describe might still be desirable

There may be some monstrous projects (millions of lines) where rewriting the code or moving it around is not feasible, so having this feature would allow them to easily integrate fpm.

In my specific case it worked:

my top-level fpm.toml

name = "<name>"
version = "0.1.0"
license = "license"
author = "<not me>"
maintainer = "<email>"
copyright = "Copyright 2022, <not me>"

[ build ]
link = ["fftw3", "blas", "lapack"]

[ library ]
source-dir = "src"
include-dir = "include"

[ install ]
library = false

[ dependencies ]
"<name1>" = { path = "include/model/<name1>" }
"<name2>" = { path = "include/param/<name2>" }

a template of a "model" and "param"

 template_model
├──  fpm.toml
├──  README.md
└──  src
    ├──  template_model.f90 # generated automatically by fpm
    ├──  file1.cu
    ├──  file2.h
    ├──  file3.f90
    ├──  file4.f90
    ├──  file5.f90
    ├──  file6.f90
    ├──  file7.f90
    ├──  file8.f90
    ├──  file9.f90
    ├──  file10.f90
    ├──  file11.f90
    └──  file12.f90

 template_param
├──  fpm.toml
├──  README.md
└──  src
    ├──  template_param.f90 # again auto generated
    ├──  file1.f90
    └──  file2.f90

[urbanjost]

Although not really what a branch is intended for, I suppose another possibility is to keep the different configurations in different git(1) branches, and then just check out the branches containing the methods you want to build with.

I can imagine a number of cases where the user-specified list of directories would make it simpler for projects already set up with Make,CMake, ... . and links are not very portable across platforms. In retrospect it seems surprising fpm(1) does not allow specifying the src directory(s).