Add how to guide for vendoring a non cmake package with ament_cmake

[hellantos]

We recently vendored an autotools package using AddExternalProject and cmake. There was not a lot of documentation out there on this and we hit on quite some issues. This guide captures our lessons learned.

[hellantos]

@clalancette Thanks for the input.

I had a look into ament_cmake_vendor_package, but it seems to only support the cmake build system for the vendored library. The library we needed uses autotools. So not all vendoring can be done with ament_cmake_vendor_package.

I would propose to update the guide with a first chapter on ament_cmake_vendor_package for cmake-based third-party libraries.

In general it would also be helpful having a document that specifies where what should be installed. I did search for that but was unsucessful finding information.

Something in the lines of: C++:

• bin/
• lib/
• share/

Python:

• bin/
• lib/pythonx.y/site-packages/...
• share/

This probably is "common knowledge" but it could be pretty useful to have that when vendoring a bit more tricky things (maybe there a rep or design document that I could not find?).

[clalancette]

I would propose to update the guide with a first chapter on ament_cmake_vendor_package for cmake-based third-party libraries.

That seems reasonable. That is, I would want the guide to suggest first using the new ament_cmake_vendor_package if it can, and then if not fall back to the alternative (which is more-or-less what you've outlined here so far).

This probably is "common knowledge" but it could be pretty useful to have that when vendoring a bit more tricky things

The honest truth is that we try to discourage vendoring in general. We know that it is sometimes necessary, but vendor packages have a whole host of problems, some of which are extremely subtle until things start failing. Also, there isn't necessarily one right way to do vendoring; how you vendor depends a lot on the situation that leads to the vendored package to begin with.

This also suggests that this guide should have a preamble that states that the ROS ecosystem discourages vendoring in general, and that it should only be done as a last resort.

[hellantos]

@clalancette @cottsay I have updated the document. To include the discussed sections.

One thing I noticed when looking at ament_vendor: did you test the RPM builds work for you with the ament_vendor() function? At least for me they fail if I do install from within ExternalProject_Add(). RPM build seems to pick installed files up as installed but not packaged files (even if installed to a build directory). You can solve it by skipping INSTALL_COMMAND in ExternalProject_Add() and then calling install(CODE "execute_process(COMMAND make install WORKING_DIRECTORY <where your make file is>)") from the main project.

[cottsay]

One thing I noticed when looking at ament_vendor: did you test the RPM builds work for you with the ament_vendor() function?

Yes, there are multiple packages passing RPM builds using ament_vendor today. The problem you're likely facing is a complex interaction between a few features in some different programs.

In short:

• A repository is cloned
• The external package is built during the vendor package's build phase
• During RPM and debian build phases, the install phase is invoked with DESTDIR
• The external project's repository is re-fetched to check for updates, which invalidates the configure and build phases of the project
• The project is re-built during the install phase, but with DESTDIR set this time, the external project's installation to the staging directory is re-routed somewhere else

The problem occurs in the debian builds as well, but it's not a critical failure like it is in the RPM builds. It's a bug either way.

[cottsay]

Also note that ament_vendor completely disables ExternalProject_Add's semi-broken repository fetch mechanism in favor of the vcstool package used frequently with ROS. So there is no "update" phase when using ament_vendor, so this isn't a problem.

One of many reasons I wrote ament_vendor to begin with.

[clalancette]

Given the lack of activity on this PR, I'm going to close it out for now. If you'd like to continue this PR and add in the suggestions we made earlier, please feel free to reopen it.