nim-lang
compiler: Option to produce a depfile from target dependencies
Title
Depfile
Abstract
Add a new command-line option (e.g. --depfile) for the nim compile command. It accepts an argument (output path for a depfile, generated for a given target).
Motivation
This change would improve Nim integrations with language-independent build systems. Particularly, get better edit-compile cycle times.
Currently a build system needs to rebuild a binary if any of source files has been changed, even if they aren't used by the binary. The full list of files that a given source file depends on can only be discovered by the compiler.
It doesn't need to be a standalone command because if the file has never been compiled, it must be built anyway, generating dependencies as a side effect.
Description
Downside: growing number of command-line flags :)
Similar proposal: #412. However custom JSON format will not be understood by build systems.
Depfile file format
From GCC manual (-M option):
The preprocessor outputs one 'make' rule containing the object file name for that source file, a colon, and the names of all the included files, including those coming from '-include' or '-imacros' command-line options.
Or, in the BNF notation:
https://cmake.org/cmake/help/latest/command/add_custom_command.html
It is recognized by all common build systems.
Code Examples
Before
(no command to produce a depfile)
After
$ nim c --depfile main.d main
$ head -4 main.d # system imports not shown
main: \
/home/project/main.nim \
/home/project/src/file1.nim \
/home/project/src/file2.nim \
Use the depfile in Makefile
main: main.nim
nim c $(NIMFLAGS) --depfile [email protected] $<
-include main.d
Use the depfile in Ninja
rule nimc:
deps = gcc
depfile = $out.d
command = nim c $nimflags --depfile $out.d $in
build main: nimc main.nim
Use the depfile in CMake
...
add_custom_command(
OUTPUT main
COMMAND nim c ${NIMFLAGS} --depfile main.d main.nim
DEPFILE main.d
VERBATIM
)
Use the depfile in Meson
...
nim = find_program('nim')
custom_target(
input : 'main.nim',
output : 'main',
depfile : '@[email protected]',
command : [nim, 'c', '--depfile', '@DEPFILE@', '@INPUT@'],
build_by_default : true
)
Backwards Compatibility
This RFC introduces no backward-incompatible changes.
So write (and contribute!) a tool that translates the json file into whatever format that you need.
So write (and contribute!) a tool that translates the json file into whatever format that you need.
I haven't found a command that generates file's dependencies. Does it exist?
Hmm. nim c --genScript:on main generates such list of files
(main.depend). So I guess a tool can be used to convert it to a
depfile and we call it a day.
However if its format was standard depfile and it was allowed to set a custom output path for that file, things would be simpler.
Nonstandard formats will not be understood by ninja / cmake / meson unless you "write (and contribute!) a tool" that both runs the compiler and converts the nonstandard format.
Makefile rules are inherently embedded shell scripts, so you can write an inline Makefile wrapper script that first runs the compiler, then runs the tool, which isn't as bad. But this doesn't work generically.
If the goal is for the compiler to support "all common build systems" then there is no alternative and no half measure -- you need the compiler itself to support the standard format.
If the goal is for "all common build systems" to specifically support the compiler, then each build system would need to have a feature request opened + accepted + implemented to have the build system support the nonstandard format. This might prove challenging.
and it was allowed to set a custom output path for that file
While it is more flexible to support a custom output path for the depfile, it isn't really required. Neither ninja nor make nor cmake nor meson require that the depfile path appear as part of the compiler rule -- they merely require that the rule ultimately creates the depfile, and that the build system knows which path to look for the depfile at. For example,
- GCC:
gcc -MD path/to/foo.c -c -o path/to/foo.odefaults to creating a depfile atpath/to/foo.d.- can be overridden by
-MF alternative/path/to/foo.o.d
- can be overridden by
- Qt moc:
--output-dep-filewhich for output files likepath/to/moc_foo.cppdefaults to createpath/to/moc_foo.cpp.d.- can be overridden by
--dep-file-path alternative/path/to/moc_foo.d
- can be overridden by
So:
- based on precedence, you do not need to have an option to specify the output file path
- strictly speaking, you do not need an option at all, you could always create a depfile unconditionally whether the user wants it or not. As long as the build system can predict where to find it. The build system won't know where to find it if it's not somewhere in the build tree, though, so if it's somewhere in $XDG_CACHE_HOME then
--nimcachemight be needed. It might be wanted regardless...
~~https://github.com/status-im/nimbus-eth2/blob/unstable/tools/generate_makefile.nim - here's an example a tool that generates makefile rules from the json~~
uninteresting, per comment below
There seems to be a misunderstanding here. The depfile needs to show a list of files that Nim read during the compilation process, not the list of compiled C sources. As a specific example, an included nim file should be part of the depfile, but it won't be mentioned at all in the json file. Slurped files, config files, etc are other build process inputs that belong to the depfile.
I've just learned that the --run option produces a proper list of depfiles stored in the nimcache json, so it looks like it just needs to be wired to a separate new option.
I've just learned that the --run option produces a proper list of depfiles stored in the nimcache json, so it looks like it just needs to be wired to a separate new option.
@zah Hi, -d:nimBetterRun exists, which produces proper depfiles and doesn't actually run the executable.
