Problems declearing local (path) dependencies

[plevold]

Description

I'm trying to organize my fpm project with multiple sub-projects, but I'm experiencing some problems which I believe is a bug or at least an ambiguity in the way fpm handles path dependencies.

Say I have the following projects:

• hierarchical (main project with app)
• mylib (library)
• utils (library)

The dependency tree is as follows: hierarchical -> mylib -> utils

I expect to be able to build and test the main project and any of the subprojects depending on what I'm currently working on. Examples:

• When adding new features to mylib I would write code and run fpm test in mylib-s directory
• When utilising the new mylib feature in the app I would write code and run fpm test in the main project directory

I've tried the following ways of organizing my code: 1)

- <repo>
    - fpm.toml (main project)
    - lib/
        - mylib/
            - fpm.toml (library)
        - utils/
            - fpm.toml (library)

- <repo>
    - hierarchical/
        - fpm.toml (main project)
    - lib/
        - mylib/
            - fpm.toml (library)
        - utils/
            - fpm.toml (library)

- <repo>
    - hierarchical/
        - fpm.toml (main project)
    - mylib/
        - fpm.toml (library)
    - utils/
        - fpm.toml (library)

See https://github.com/plevold/fpm-path-bug for the complete set of files.

This is what happens with the different folder layouts:

• Case 1:
  • Build and test mylib: Works fine
  • Build and test main project: Fails with <ERROR>*cmd_run*:model error:'.\../utils\fpm.toml' could not be found, check if the file exists
• Case 2:
  • Build and test mylib: Works fine
  • Build and test main project: Fails with <ERROR>*cmd_run*:package error:'fpm.toml' could not be found, check if the file exists
• Case 3:
  • Build and test mylib: Works fine
  • Build and test main project: Works fine

Expected Behaviour

I would expect all of these project layouts to work.

It seems like what's happening is that fpm resolves the paths relative to the fpm.toml file being built and not the fpm.toml of the path-dependency. This gives different behaviour when I try to build the main project or mylib. In the third case, the relative path from the main project to utils and from mylib to utils is "by accident" the same (i.e. ../utils).

I think fpm should resolve paths relative to the current fpm.toml file being parsed and not the first one. This would make all the different project layouts in my examples work.

As a side note there should maybe be a recommended way of organizing projects like this. I'm not sure which way is better, but that's an interesting discussion for sure 😄

Version of fpm

0.4.0

Platform and Architecture

Windows

Additional Information

No response

[urbanjost]

If the two libraries and the application are developed independently, the approach is assumed that there are three top project directories, and that the app uses the two other projects as dependencies. The dependencies can be links to the pathnames instead of URLs.

Or, you just have one fpm.toml at the top of the project, and named tests in the test/ directory that you would call, such as

fpm test my_utils

There could have been changes, but fpm(1) was not initially designed to include subprojects in the same directory except as dependencies. An interesting approach that others might feel should be implemented, but the two schemes described above provide the same functionality, essentially. Which to use mostly depends on whether you want to independently compile and/or reuse the libraries, or want everything to build at once with the same options.

There are ways to have the file layout you describe with everything after lib/ but you have to turn off auto-search for sources and explicitly name the files to build; the fact you put files into lib/ instead of src/ means you did something in your fpm.toml file that we need to see if you want to stick with that layout.

[urbanjost]

As an example of creating three directories (on a *nix system with bash, but the concept works on all supported platforms) and using directory B and C as dependencies of A (A and B do not have to be links, they could be subdirectories contained in A)

#!/bin/bash
# assuming you are building library B in directory B and library C in 
# directory C and your directory with an application is in A
fpm new A
fpm new B
fpm new C
cd A
# make links to the other projects
ln -s ../B B
ln -s ../C C
###############################################################################
# declare the other directories to be dependencies
cat >>fpm.toml <<\EOF
[dependencies]
B      = { path = "B" }
C      = { path = "C" }
#B      = { git = "https://github.com/urbanjost/B.git" }
#C      = { git = "https://github.com/urbanjost/C.git" }
EOF
###############################################################################
# make a program that calls the other projects
cat >app/main.f90 <<\EOF
program main
  use A, only: helloa => say_hello
  use B, only: hellob => say_hello
  use C, only: helloc => say_hello
  implicit none

  call helloa()
  call hellob()
  call helloc()
end program main
EOF
###############################################################################
fpm run
exit

The output should be

 Hello, A!
 Hello, B!
 Hello, C!

[urbanjost]

I think these steps would be closest for what you were originally entertaining:

myproject
    ├── app
    │   └── main.f90
    ├── fpm.toml
    ├── lib
    │   ├── mylib
    │   │   ├── fpm.toml
    │   │   ├── README.md
    │   │   ├── src
    │   │   │   └── mylib.f90
    │   │   └── test
    │   │       └── check.f90
    │   └── utils
    │       ├── fpm.toml
    │       ├── README.md
    │       ├── src
    │       │   └── utils.f90
    │       └── test
    │           └── check.f90
    ├── README.md
    └── test
        └── check.f90

#!/bin/bash
fpm new myproject --test --app
cd myproject
# add dependencies
cat >>fpm.toml <<\EOF
[dependencies]
mylib      = { path = "lib/mylib" }
utils      = { path = "lib/utils" }
EOF
# make program that uses dependencies
cat > app/main.f90 <<\EOF
program main
    use utils, only: hellob => say_hello
    use mylib, only: helloc => say_hello
    implicit none
   
    call hellob()
    call helloc()
end program main
EOF
# make the subprojects
fpm new utils --src --test
fpm new mylib --src --test
mkdir lib
mv utils mylib lib/
# run app
fpm run

[plevold]

the fact you put files into lib/ instead of src/ means you did something in your fpm.toml file that we need to see if you want to stick with that layout.

@urbanjost as mentioned, please see https://github.com/plevold/fpm-path-bug for the complete set of files.

In the last example you show your dependency tree looks like this:

myproject -> mylib
          -> utils              

In the example I show the main project does not depend on utils, but mylib does. I.e. the dependency tree looks like this:

myproject -> mylib -> utils

This triggers the error while your example does not.

[LKedward]

cc @awvwgk

Thanks for reporting @plevold. The manifest reference says this about local path dependencies:

Local dependency paths are given relative to the project root and use / as path separator on all platforms.

It's not clear which 'project root' that refers to but it would seem that, perhaps counter-intuitively, all such local paths are currently required to be relative to the top-level project root. Of course, this means that the manifests of the intermediate dependencies (e.g. mylib) are non-sensical in the absence of the top-level project.

As shown here, this behavior needs changing because the local paths should be relative to the manifest file which defines them and not the top-level project. Would you agree @awvwgk?

I believe the relevant implementation is in fpm_dependency.f90 but I don't know yet what changes are required.

[epagone]

Sorry to chime-in, please hide this message if too much off-topic.

If some re-design of the local dependencies implementation will be done, I will ask to consider to allow "out-of-tree" local dependencies. E.g. I have a more generic library gen-lib as fpm package that is not pushed to the repository of a remote version control system but that is used by several local projects app1, lib1, app2 in this way

gen-lib
    ├── src
    │   └── *.f90
    └── fpm.toml

app1
    ├── app
    │   └── *.f90   <--- use gen-lib
    └── fpm.toml

lib1
    ├── src
    │   └── *.f90   <--- use gen-lib
    └── fpm.toml

app2
    ├── app
    │   └── *.f90   <--- use gen-lib
    └── fpm.toml

In this way I can centrally work on gen-lib and then cascade the amendments to app1, lib1, app2, etc...

I understand that the end result might be a copy or a link of the "out-of-tree" dependencies in the project tree, but it would be nice to make this transparent to the fpm user.

[zoziha]

I'm having the same problem, the path is always relative to the top-level project, I tried reading Cargo's help documentation, the dependency path should be relative to the Cargo.toml file that contains it, which makes more sense. (Maybe we can spend some time trying to figure this out.)

(See link) This tells Cargo that we depend on a crate called hello_utils which is found in the hello_utils folder (relative to the Cargo.toml it’s written in).