Circular exec_depend not possible

[luator]

System Info

• Operating System: Linux SNCN0051 4.4.0-135-generic #161-Ubuntu SMP Mon Aug 27 10:45:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

• Python Version: Python 2.7.12

• Version of catkin_tools:

  catkin_tools 0.4.4 (C) 2014-2018 Open Source Robotics Foundation
  catkin_tools is released under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)
  ---
  Using Python 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]
  

• ROS Distro: kinetic

Build / Run Issue

catkin build fails when I have two packages foo and bar with a circular exec_depend.

• [x] Works with catkin_make
• [x] Works with catkin_make_isolated --merge
• [ ] Works with catkin build
• [ ] Works with catkin build -p1
• [ ] I did not read this

Expected Behavior

According to my understanding, exec_depend means, the dependency is only needed when executing something of the package. So during build, exec_depend should be completely irrelevant and thus a circular exec_depend should not be a problem for catkin build.

Actual Behavior

❱ VERBOSE=1 catkin build -p1 -v
...                                                                                                                                                   
Traceback (most recent call last):
  File "/home/fwidmaier/src/catkin_tools/catkin_tools/verbs/catkin_build/build.py", line 549, in build_isolated_workspace
    continue_without_deps=False))
  File "/home/fwidmaier/src/catkin_tools/catkin_tools/execution/executor.py", line 374, in run_until_complete
    return loop.run_until_complete(coroutine)
  File "/usr/lib/python2.7/dist-packages/trollius/base_events.py", line 350, in run_until_complete
    return future.result()
  File "/usr/lib/python2.7/dist-packages/trollius/futures.py", line 285, in result
    compat.reraise(type(self._exception), self._exception, exc_tb)
  File "/usr/lib/python2.7/dist-packages/trollius/tasks.py", line 259, in _step
    result = coro.send(value)
  File "/usr/lib/python2.7/dist-packages/trollius/tasks.py", line 387, in wait
    raise ValueError('Set of coroutines/Futures is empty.')
ValueError: Set of coroutines/Futures is empty.

Complete output in gist: https://gist.github.com/luator/dd1d0d9112dfebfc734a06d9900054b0

logs: logs.tar.gz

Steps to Reproduce the Issue

Using this minimal workspace (contains two empty package that exec_depend on each other): src.tar.gz

catkin build

[luator]

Related to this (not sure if it is the same issue or a separate one): When one of the packages has exec_depend and the other a build_depend, I would still expect that it should work (since only build_depend should be relevant at build time. In this case, however, I get the following error:

[build] Error: The workspace packages have a circular dependency, and cannot be built. Please run `catkin list --deps` to determine the problematic package(s).

To reproduce, take the example workspace from above and change the dependency of foo to bar from exec_depend to build_depend. Note, however, that in this case also catkin_make fails.

[mikepurvis]

Previously: #439, #440, #429

Fix is already merged as of e30c226276c657687942e9ec48e6596175aafaa5; but it isn't in the latest release (0.4.4). In the meantime, consider running catkin_tools from source.

[luator]

@mikepurvis I just tried with the current master and get exactly the same issue. Is my example workspace working for you?

[ianjfrosst]

Tried this with the latest release (0.4.5), and could reproduce the issue with the example workspace. Can someone (@mikepurvis?) clarify why exec_depend packages are even being considered when building?

[wjwwood]

@ianjfrosst if you don't consider exec depends, then you cannot know that you can run (or test) a package after building it.

[mikepurvis]

The logic is supposed to be that catkin build foo yields a resultspace where all of foo's dependencies that are available in the source space get built, with only build_depend considered for the purposes or ordering the build.

Obviously the problem remains, though, even after e30c226276c657687942e9ec48e6596175aafaa5, which resolved the problem in my scenario and specifically adds a unit test to cover the mutual run/exec depend case. :) I'd be delighted for someone to take a look into what's going on here.

[ianjfrosst]

@mikepurvis that logic is what I expected, but it's not what currently happens. If you have a workspace with foo and bar, and foo has <exec_depend>bar</...>, both catkin build and catkin build foo will build bar before foo. @wjwwood the issue is that exec_depend is causing package builds to be serialized. If you catkin build a workspace, you could build a package and its exec depends at the same time, and still end up with everything you need to run a package after building it.

[wjwwood]

@wjwwood the issue is that exec_depend is causing package builds to be serialized. If you catkin build a workspace, you could build a package and its exec depends at the same time, and still end up with everything you need to run a package after building it.

That makes sense to me now, thanks. I agree that they should be parallelized.

However, I was reacting to the original question about circular dependencies, in which case the solution is not to ignore exec depends, but to not use them in topological ordering.

---

@ianjfrosst When you said:

Can someone (@mikepurvis?) clarify why exec_depend packages are even being considered when building?

I understand "building" to mean starting when catkin build starts and ending when it finishes, in which case you certainly do need to consider the exec depends.

I think what would have been clear to me, is if you had asked why exec depends were being considered in topological ordering (which dictates in what order things are built and whether or not they can be parallelized).

[wjwwood]

However, I would recommend to everyone to avoid circular dependencies, exec or not. It just makes your life easier. I'm not aware of a use case that requires them and has no architectural alternative.

Group dependencies might be helpful for some cases:

http://www.ros.org/reps/rep-0149.html#group-dependencies

It's part of the reason why we did package.xml format 3.