Support custom build script

[LKedward]

Match behaviour of bootstrap version

[awvwgk]

I would prefer to restructure the build-script logic in the package manifest first, see #249, before implementing the logic in Fortran fpm.

[ivan-pi]

I could not find any previous discussion pertaining to the bootstrap fpm design of the build-script logic. It seems to be something that @everythingfunctional built in initially to get the project off the ground?

Edit: I found a former discussion in this issue: https://github.com/fortran-lang/fpm/issues/118

Do any of the libraries used to bootstrap fpm actually require a custom build script? If not, I don't see any reasons why not to restructure the build script logic in the manifest. Since fpm is still in an alpha stage, I don't think this will cause many problems for users.

[everythingfunctional]

I did indeed come up with the initial design primarily on my own. I was trying be compatible with what I believed to be common best practice in Makefiles, but we did discuss some changes that probably ought to be made. Primarily I think we concluded that all "inputs" to the build scripts should be through environment variables prefixed with FPM_, and any information the script would like to report back to fpm would be via stdout on lines prefixed with fpm:. This is similar to Cargo's design, and I agree it seems to be working out pretty well for them.

I'm open to suggestions on how such scripts should be specified and executed in fpm.toml. Mine was just a first draft.

[ivan-pi]

Thanks @everythingfunctional for the explanation. I suggest we shift to #249 to figure out a specification.

[ivan-pi]

The discussion in #249 suggested we separate between one-way and two-way build scripts. This issue is for one-way build scripts.

The suggestion from @awvwgk was to use the following manifest syntax:

build.script = ["make", "-f", "build.mk"]  # or just ["build.mk"], make extension is detected
build.script = ["sh", "build.sh"]  # or just ["build.sh"], shell extension is detected
build.script = ["python", "build.py"]  # or just ["build.py"], Python extension is detected
build.script = ["ruby", "build.rb"] # or just ["build.rb"], Ruby extension is detected
build.script = ["cmake"]  # we might detect that it is CMake and run multiple steps for this script

The one-way build script is expected to receive inputs via environment variables, and place the compiled libraries in the correct location.

I think the following points from @everythingfunctional in https://github.com/fortran-lang/fpm/issues/68#issuecomment-622112176 summarize what was the former Haskell implementation:

The end result of building a library in FPM is just a .a file, and all of the relevant .mod files. So, if your package specifies a build script for that, FPM will just call it. There is a small set of things that FPM would like to dictate to that script though. Those being:

1. The compiler to use
2. The compiler flags to use (mostly to ensure the flags are compatible with the given compiler)
3. Where to put the archive and module files
4. Where to find any of the dependencies

Regarding point 4, I think it should be excluded from one-way build scripts.

A few questions that come to my mind:

• Where should the build script be located within the project tree?
• Should all the environment variables consumed by the build script be prepended with FPM_?
• How to communicate build profiles like debug or release? (Are these propagated through the compiler flags by an environment variable like FPM_FFLAGS or would we simply pass a variable called PROFILE like Cargo does)
• Where should the archives be placed? Should it be a sub-folder in /build/<compiler>_<profile>/ meaning the build script would receive an environment variable with the output folder name? Cargo uses the variable OUT_DIR for this purpose.
• Should the "acceptable" build script outputs be limited to .a/.lib for C/C++/Fortran archives, .mod for Fortran modules, and .h/.hpp for C headers? Would it be acceptable to link in object files directly?

[everythingfunctional]

Where should the build script be located within the project tree?

The Haskell implementation assumed the top level of the project. I would initially lean towards that, but if there's a possibility that we want to support build scripts for executables and/or tests, the perhaps it should go in src-dir.

Should all the environment variables consumed by the build script be prepended with FPM_?

I'd have to go back to the discussion to be sure, but IIRC we were leaning towards yes.

How to communicate build profiles like debug or release? (Are these propagated through the compiler flags by an environment variable like FPM_FFLAGS or would we simply pass a variable called PROFILE like Cargo does)

I would lean towards starting with just FPM_FFLAGS. I'm not sure a build script would need to now about profiles, but I'm open to being convinced.

Where should the archives be placed? Should it be a sub-folder in /build/<compiler>_<profile>/ meaning the build script would receive an environment variable with the output folder name? Cargo uses the variable OUT_DIR for this purpose.

I think yes, the script receives an environment variable specifying the output folder name. I used BUILD_DIR in the Haskell implementation. I think for consistency sake it ought to be prefixed with FPM_, so my vote would be for FPM_BUILD_DIR. I think FPM_OUT_DIR is accpetable too.

Should the "acceptable" build script outputs be limited to .a/.lib for C/C++/Fortran archives, .mod for Fortran modules, and .h/.hpp for C headers? Would it be acceptable to link in object files directly?

I think it would be perfectly fine for a script to produce more outputs if it wanted too. So long as it produces the necessary library and module files for other Fortran sources to make use of it everything should be fine. Anything else produced probably shouldn't be looked at or relied upon by anything else, but I don't think that's something worth trying to enforce.

[ivan-pi]

Where should the archives be placed? Should it be a sub-folder in /build/<compiler>_<profile>/ meaning the build script would receive an environment variable with the output folder name? Cargo uses the variable OUT_DIR for this purpose.

I think yes, the script receives an environment variable specifying the output folder name. I used BUILD_DIR in the Haskell implementation. I think for consistency sake it ought to be prefixed with FPM_, so my vote would be for FPM_BUILD_DIR. I think FPM_OUT_DIR is accpetable too.

BUILD_DIR sounds good too.

What happens if my project depends on two packages, which export a library with the same name? Can this cause a linking problem?

Does the library generated by the one-way build script need to be specified manually under the link key in the [build] table?

[everythingfunctional]

What happens if my project depends on two packages, which export a library with the same name? Can this cause a linking problem?

Does the library generated by the one-way build script need to be specified manually under the link key in the [build] table?

I think this is why fpm should dictate the name of the library to be produced. You already can't depend on two packages with the same name, and fpm uses the name of the package to determine the name of the library. Thus, fpm should dictate to a build script the name of the library file to be produced.

With the above, no, you don't need to manually specify the library under the link key in the same way your don't need to specify one for a package that doesn't have a build script. As far as users of your package are concerned, they shouldn't have to care or even know that it's built with a custom script.

[ivan-pi]

Would this imply the convention that a simple build script only produce one library (archive)? Or are you referring here to an fpm package as it's own entity.

Concerning your second paragraph, I was not concerned about users of the package. I know this is handled by fpm already. In my current mental model I assumed that the build script is allowed to generate any number of lib<name>.a files. The package maintainer then adds these to the link = ["liba","libb","libc",...] section. Fpm takes care of the rest.

Addendum: I think I understand your view. I'm guessing that in the haskell-fpm, the implicit rule was that the package and the library produced by the build script share the same basename?

[everythingfunctional]

For a one way build script, fpm wouldn't know that other libraries are produced. I'm guessing we're already including different build directories as include locations in the link commands, so manually adding those additional libraries to the link works, but I'm not sure it was intended to.

I would argue that a package that produces more than one library should be split into separate packages. I'd say we shouldn't focus any effort trying to explicitly support packages that want to produce multiple library files. You're solution is a perfectly valid workaround (if not explicitly supported), and so we shouldn't try to prevent it, just say it's not guaranteed to be reliable.

[zoziha]

The one-way build script is expected to receive inputs via environment variables, and place the compiled libraries in the correct location.

I have a good idea, it should be able to use different make tools, such as make, cmake.

1. I think the make tool generally only displays simple commands, such as make build and cmake build. Obviously, these are commands and we need to deal with them. They are more like coherent tasks✨ besides static commands.
2. Use the binary files generated by the make tool, such as link libraries and intermediate .obj files. Only the make tool and developers know where they are generated.

Based on the above analysis, we can add items like [make] to fpm.toml:

[package]
name = "fpm-make-test"

[make]
[make.tasks.src]
description = "Generate src_dir objs."
command = "make"
args = ["-f", "makefile", "--directory=src"]
kind = "objs"     # shared/static/objs/binary
objs_dir = ["./build/objs/src1/", 
            "./build/objs/src2/",
            ...
]

[make.tasks.others]
...

We use fpm's absolute control over fpm.toml to control the commands of the make tool and the path of the generated binary file. fpm selects the behavior of fpm by extracting the path of the generated binary file.

We leave this [make] to the developer to consider. fpm just sends make commands and accepts task results. My inspiration comes from:

1. https://medium.com/@sagiegurari/automating-your-rust-workflows-with-cargo-make-part-1-of-5-introduction-and-basics-b19ced7e7057
2. https://github.com/sagiegurari/cargo-make

(also see https://github.com/fortran-lang/fpm/issues/118#issuecomment-851415588)