Unable to add include paths to headers generated with custom targets that output directories

[lucasdemarchi]

My use case is to include generated headers from mavgen.py (mavlink generator) script. It generates headers on subdirectories and I can generate them with the following snippet:

mavgen =  find_program('modules/mavlink/pymavlink/tools/mavgen.py')
mavgen_wrapper = find_program('tools/mavgen.sh')

gen_mavlink_headers = custom_target('gen_mavlink_headers',
                                    input : ['modules/mavlink/message_definitions/v1.0/ardupilotmega.xml'],
                                    output : ['mavlink_headers.done'],
                                    command : [mavgen_wrapper, mavgen, '@INPUT@', '@OUTPUT@'])
mavlink_includes = include_directories('mavlink/ardupilotmega')

I use a wrapper script that just does a touch on the output file afterwards.

However checking the generated build.ninja file, the places that use mavlink_includes as include include only the src directory, not the build directory. If I do any change on the meson.build file and re-run ninja, then it works.

Checking the source code, the following diff fixes it for me, but doesn't look to be the right fix:

diff --git a/mesonbuild/backend/ninjabackend.py b/mesonbuild/backend/ninjabackend.py
index e8fae8e..cd4b821 100644
--- a/mesonbuild/backend/ninjabackend.py
+++ b/mesonbuild/backend/ninjabackend.py
@@ -1934,16 +1934,7 @@ rule FORTRAN_DEP_HACK
                 # Add source subdir first so that the build subdir overrides it
                 sargs = compiler.get_include_args(srctreedir, i.is_system)
                 commands += sargs
-                # There may be include dirs where a build directory has not been
-                # created for some source dir. For example if someone does this:
-                #
-                # inc = include_directories('foo/bar/baz')
-                #
-                # But never subdir()s into the actual dir.
-                if os.path.isdir(os.path.join(self.environment.get_build_dir(), expdir)):
-                    bargs = compiler.get_include_args(expdir, i.is_system)
-                else:
-                    bargs = []
+                bargs = compiler.get_include_args(expdir, i.is_system)
                 commands += bargs
             for d in i.get_extra_build_dirs():
                 commands += compiler.get_include_args(d, i.is_system)

Any ideas? Thanks.

[lucasdemarchi]

This is the project with the additional commit to add meson: https://github.com/01org/mavlink-router/tree/meson

[rhd]

I believe you have to list all of the output files in this line: output : ['mavlink_headers.done']. Otherwise, meson has no idea what to track.

[lucasdemarchi]

The headers are in subdirectories and meson doesn't allow me to add subdirectories in the output.

[nirbheek]

Ordinarily, the way to do this is that you would add the custom target that generates the headers as sources of the build targets that use them. Then Meson will automatically add the directory into which the headers were generated to the include paths of the build target. Something like this:

[in some source dir]
h = custom_target('gen-headers',
                  input : 'in.xml',
                  output: ['out1.h', 'out2.h', ...],
                  # Runs `tool in.xml out1.h out2.h`
                  command : ['tool', '@INPUT@', '@OUTPUT@'])
...
[in some other source dir]
executable('uses-header', 'foo.c', h, 'bar.c')

It looks like in your case you are indeed doing that, but your tool outputs in a subdirectory. Can you change that to generate in the current directory?

[lucasdemarchi]

@nirbheek I can tell it to generate in the current directory, but that changes how to include the headers and I don't think it will help. mavgen tool generates 209 headers and will use subdirs even if the root is not in a subdir. And I can't add subdir/header_in_subdir.h to the output of custom_target().

Normally one uses mavlink as the directory where headers are generated so you use #include <mavlink/mavlink.h>, #include <mavlink/checksum.h> and so on... Having just #include <checksum.h>, without the subdir, would create lots of problems...

See pymavlink/tools/mavgen.py -o /tmp/mavlink --lang C message_definitions/v1.0/ardupilotmega.xml: https://gist.github.com/7eb74069da9125832e5f519017a3ee2f

I tried to understand why the check of the directory existence is in the place I mentioned, but I failed to find out why.

[nirbheek]

If you want to #include <mavlink/foo.h>, then using output: 'mavlink' (the directory) and @OUTDIR@ in the command: should work for you. You can skip the wrapper shell script.

[nirbheek]

Sorry, not @OUTDIR@, but @OUTPUT@ in command:.

[rhd]

@nirbheek if you only specify the directory in the output, then meson will only track the existence of that directory and nothing inside of it, right? Then somehow you'd have to convert that directory to an include_directories?

maven_dir = custom_target('gen-headers',
                  input : 'in.xml',
                  output: ['maven'],
                  # Runs `tool in.xml out1.h out2.h`
                  command : ['tool', '@INPUT@', '@OUTPUT@'])

executable('uses-header', 'foo.c',
    include_directories: include_directories(maven_dir),
)

[nirbheek]

Well, like I said, Meson should take care of adding the include directories for you as long as you add that custom target to the list of sources. You should not need to use include_directories() at all.

[rhd]

Hi @nirbheek, so you're saying that if you do:

maven_dir = custom_target('gen-headers',
                  input : 'in.xml',
                  output: ['maven'],
                  command : ['tool', '@INPUT@', '@OUTPUT@'])
executable('uses-header', 'foo.c', maven_dir)

meson will add -Imaven to the gcc command line? Or does custom_target return all of the files inside of the maven directory after the command gets executed? Or am I totally missing something.

@lucasdemarchi sorry for hijacking the issue

[nirbheek]

# In geninc/meson.build
maven_dir = custom_target('gen-headers',
                  input : 'in.xml',
                  output: ['maven'],
                  command : ['tool', '@INPUT@', '@OUTPUT@'])

and

# In meson.build
executable('uses-header', 'foo.c', maven_dir)

Meson will add -Igeninc while compiling foo.c because it knows you want to use those headers in the target uses-header. Meson does not care what the output is; whether it's a file, a directory, or several files. If it can't be classified as a source file, it will not be used as a source.

[lucasdemarchi]

If you want to #include <mavlink/foo.h>, then using output: 'mavlink' (the directory) and @OUTDIR@ in the command: should work for you. You can skip the wrapper shell script.

This doesn't seem to work. It tries to include mavlink from the source dir (where it doesn't exist) rather than on the build dir. This is the command it uses:

FAILED: mavlink-routerd@exe/endpoint.cpp.o 
ccache c++  -Imavlink-routerd@exe -I. -I.. -I../mavlink/ardupilotmega -I../mavlink -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -std=gnu++11 -O0 -g -D_GNU_SOURCE -DPACKAGE_VERSION=1 -MMD -MQ 'mavlink-routerd@exe/endpoint.cpp.o' -MF 'mavlink-routerd@exe/endpoint.cpp.o.d' -o 'mavlink-routerd@exe/endpoint.cpp.o' -c ../endpoint.cpp
In file included from ../endpoint.cpp:18:0:
../endpoint.h:20:21: fatal error: mavlink.h: No such file or directory

Taking a look again, it's not true that source code does #include <mavlink/mavlink.h>. It actually does only "#include <mavlink.h>". So what I want is the following part to include from the build dir, not the source dir (where it doesn't make sense):

mavlink_includes = include_directories('mavlink', 'mavlink/ardupilotmega')

[lucasdemarchi]

It seems the problem is that it's not adding the include from the build directory. I pushed something similar to the suggested changes here: https://github.com/01org/mavlink-router/commits/meson

[nirbheek]

We can't revert those changes because they cause build failures for people with newer, stricter compilers and -Werror. Can you try this patch? It's a bit hacky but it should work.

I am not sure how we would make this easier to do in Meson. Will have to think about it.

[lucasdemarchi]

@nirbheek I saw the patch when you sent and was going to test it now, but it's not there anymore. As far as I remember from the patch, there are some issues:

Why does the srcdir need to have the same structure as the builddir? I certainly don't want a "ardupilotmega" directory on the root of my repository.

Would a flag to "include_directories()" marking it as "builddir-only" be ok? It would default to false, not breaking existing use cases and then we a) don't add srcdir and b) always call bargs = compiler.get_include_args(expdir, i.is_system) in the snippet above.

[lucasdemarchi]

We can't revert those changes because they cause build failures for people with newer, stricter compilers and -Werror. Can you try this patch? It's a bit hacky but it should work.

So the issue is https://github.com/mesonbuild/meson/issues/1185 and the solution is b01d2c35b7003071fc64c6d2e7d07b7652a8b386 as far as I could check. I.e. it tries not to put the include dir from the build directory due to the use of -Wmissing-include-dirs. However this means the command to run will depend on the structure of the build dir at the time it's running. This seems wrong and causes the issue detailed here:

$ meson build && cd build && ninja # fails
$ meson build && cd build && ninja && touch ../build.meson && ninja # works

I'm taking a look on providing the solution I said above, but to me the previous change looks wrong.

[nioncode]

Any progress on this? I like the solution @nirbheek proposed in #2259: let custom_target (and generators?) take a new kwarg header_include (or creates_include to make it even more explicit?) that takes an array of directories that will be added to the include_directories of consumer's.

If nobody implements this, feel free to assign me.

[jpakkane]

Targets already have a private_dir_include method which should provide for this.

[nioncode]

Does this work for generators and custom_target? The documentation only mentions executables, libraries, and modules. Also, how should this property be populated for generators and custom_target? It must be derived from the output or given explicitly.

[jpakkane]

For generators you'd use the custom target object's private dir include. Generator outputs do not live anywhere on their own (at least they don't at the moment).

[marler8997]

I've run into this problem at work (I work at HP). I've been trying to get us to use meson rather than CMake, but not having a solution to this is making it quite hard to convince people. Or maybe I'm just doing something wrong? I've simplified the problem, created an app/library example with the goal of being able to use the library if it's installed, or fallback to downloading/building it from source if it's not. I use a custom_target to download the source in this case, which means I am running into the problem in this issue with header files.

I've created a repo that demonstrates the issue. In it I've included a theoretical meson syntax and explain the semantics I would expect to be able to do, however, maybe someone knows a better way to accomplish what I'm trying to do? Pull requests and guidance are welcome. Thanks.

https://github.com/marler8997/meson-example-app

[jpakkane]

I've simplified the problem, created an app/library example with the goal of being able to use the library if it's installed, or fallback to downloading/building it from source if it's not.

Meson provides builtin functionality for this already. If you can use it instead of rolling your own, it will simplify things a lot. See for example this, [this]{https://mesonbuild.com/Wrap-dependency-system-manual.html) and related documentation.

[marler8997]

@jpakkane Thanks for the guidance here. I was able to use the wrap feature in this case nicely.

That being said, we also have tools that generate header files which will still be affected by the issue described in this thread. Namely, adding include directories from custom_target outputs. Is there any solution to it in the forseeable feature? Would the solution I provided in my meson-example-app be a viable solution? Namely, adding a function to custom target objects to get their output objects? i.e.

header_generator = custom_target(
  # assume this target generates headers in the output directory
  output : ['generated_include_dir'],
  ...
)

executable(...,
  include_directories : [header_generator.output(0)],
)

And another example, where the generated headers are in a sub-directory of the output. In this case I thought you could add a method to directory objects that allow you to get relative paths from them (relpath), i.e.

source_and_header_generator = custom_target(
  # assume this target generates sources and headers in the 'inc' subdirectory
  output : ['generated_dir'],
  ...
)

executable(...,
  source_and_header_generator,
  include_directories : [source_and_header_generator.output(0).relpath('inc')],
)

Thanks in advance for your time on this.

[jpakkane]

Instead of having the headers in an inc dir, have the generator write them to the build dir. If it can not do that then invoke it via a wrapper script that does the fixup. Then you can do something like:

gen_headers = custom_target(...,
    output: ['file1.h', 'file2.h', ...],
    command: [copier_wrapper_script, ...])
gen_inc = include_directories('.')

and then do this in the target that needs those headers:

executable(..., gen_headers,
    include_directories: [gen_inc, ...])

[RisinT96]

Are there any updates? I'm also trying to use the mavgen.py, the problem is that it generates around 200 headers, in a specific directory structure, all from a couple XMLs.

Also depending on the content of the XMLs, it might generate different headers (different filenames).

Currently I have some workaround but it's very ugly, and if the xml is changed, I have to compile once to regenerate the headers, and compile again to detect that the headers have changed.

This is my current solution:

mavgen =  find_program('submodules/pymavlink/tools/mavgen.py')
gen_mavlink_headers = custom_target(
  'gen_mavlink_headers',
  input : ['src/pilot/mavlink/message_definitions/ardupilotmega.xml'],
  output : ['mavlink.stamp'],
  command : [
    mavgen,
    '--lang',
    'C',
    '--wire-protocol',
    '2.0',
    '-o',
    # Generates the headers inside the source dir, will see why in a moment.
    '@SOURCE_DIR@/src/pilot/mavlink/includes',
    '@INPUT@',
    '&&',
    'touch',
    '@OUTPUT@'
  ]
)

includes = [
  # Can't add these in the build dir... as it doesn't exist.
  include_directories(
    'src/pilot/mavlink/includes',
    'src/pilot/mavlink/includes/ardupilotmega',
    'src/pilot/mavlink/includes/common',
  )
]

project_target = executable(
  meson.project_name(),
  project_source_files,
  gen_mavlink_headers,
  include_directories: includes,
  dependencies: project_dependencies,
  cpp_args : build_args,
)

Had to create dumy files to have git keep the folder structure.

There must be a better way!

Please help :)

[dcbaker]

Okay, I've been thinking about this problem a lot. @jpakkane what if we allowed custom_target to write directories in its private directory?

[lucasdemarchi]

@dcbaker does that mean it would accept a directory as output? But then what would prevent meson to add e.g. -Isrc/non-existent-directory and break the build?

[dcbaker]

Yes, that would allow directories as part of the output, in general with custom targets you don't want to pass -I with custom_target anyway, meson will generate a -I for you when you add it as a source. We would need to work exactly what the semantics of adding a custom_target with directory outputs and headers would mean.

[lucasdemarchi]

@dcbaker the main problem here is that currently it will add a -I for the source directory and another for the build directory. See
https://github.com/mesonbuild/meson/pull/2259#issuecomment-325765576 - we'd need to change that semantic for custom_target.

[lucasdemarchi]

@dcbaker did you have a chance to try your solution?