Catkin tool build issues with symlinking

[tiusty]

System Info

• Operating System: Linux alexDev-VirtualBox 4.13.0-26-generic #29~16.04.2-Ubuntu SMP Tue Jan 9 22:00:44 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Python Version: Python 2.7.12
• Version of catkin_tools: catkin_tools 0.4.4 (C) 2014-2018 Open Source Robotics Foundation catkin_tools is released under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)

---

Using Python 2.7.12 (default, Nov 20 2017, 18:23:56) [GCC 5.4.0 20160609]

• ROS Distro: kinetic

Build / Run Issue

• [x] Works with catkin_make
• [x] Works with catkin_make_isolated --merge
• [ ] Works with catkin build
• [ ] Works with catkin build -p1
• [x] Works with catkin build -v
• [x] Works with VERBOSE=1 catkin build -p1 -v

Expected Behavior

The expected behavior is for all the packages to build successfully and create a tools directory that contains all the executables that we built.

Actual Behavior

When the build is run with either: catkin build or catkin build -p1 the build doesn't work but when the build is run with catkin build -v then the build successfully builds or others as noted above.

Therefore my main question is, what is the difference between catkin build and catkin build -v where the verbose version of the build works (can build the software successfully) but the non-verbose version doesn't.

Some background on the error.

1. We are building 4 executables
2. The 4 source files and the make files are copied into the build space
3. Then the CMakeLists.txt will run 'add_custom_target' to executable the makefiles pointing to the source in the build directory.
4. The Makefiles will then build the executables to the devel directory.

With the working version: The executables are built in the devel/.private directory and then afterwards it is symlinked to the merged devel space devel/.

With the broken version (catkin build or catkin build -p1): The executables are built in devel/.private directory but fails to symlink them to the merged devel workspace. Instead it throws an error: No handlers could be found for logger "trollius"ed] [fly_tools:make (50%) - 2.3] [shared_interfaces:cmake - 2.2]

And then later it gives the traceback:

[build] An internal error occurred!                                                                                                                                                                        
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/catkin_tools/verbs/catkin_build/build.py", line 549, in build_isolated_workspace
    continue_without_deps=False))
  File "/usr/lib/python2.7/dist-packages/catkin_tools/execution/executor.py", line 370, in run_until_complete
    return loop.run_until_complete(coroutine)
  File "/home/alex/.local/lib/python2.7/site-packages/trollius/base_events.py", line 350, in run_until_complete
    return future.result()
  File "/home/alex/.local/lib/python2.7/site-packages/trollius/futures.py", line 285, in result
    compat.reraise(type(self._exception), self._exception, exc_tb)
  File "/home/alex/.local/lib/python2.7/site-packages/trollius/tasks.py", line 259, in _step
    result = coro.send(value)
  File "/usr/lib/python2.7/dist-packages/catkin_tools/execution/executor.py", line 166, in async_job
    stderr=logger.get_stderr_log(),
  File "/usr/lib/python2.7/dist-packages/catkin_tools/execution/io.py", line 119, in get_stderr_log
    return self._decode(self.stderr_buffer)
  File "/usr/lib/python2.7/dist-packages/catkin_tools/execution/io.py", line 131, in _decode
    return data.decode('utf-8')
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 3343: invalid start byte

Unfortunately I cannot give the source code to reproduce the error but hopefully there is some information to lead me to help figure out the issue.

1. Do you know why catkin build -v would work and catkin build won't?
2. The executable is built but it fails to symlink so it seems like a python issue? When i go to the .private directory I can see the generated executable from the makefile even though the build fails
3. Is there any other way to see more verbose catkin build output besides catkin build -v since that makes the build succeed?

[wjwwood]

If catkin build -v works, but without -v doesn't then I'd say it has to be a race condition. Does it work if you just run catkin build again, i.e. do repeated runs fails?

[tiusty]

Here is a sequence of buildings:

sequence 1: catkin clean -y catkin build (fails) catkin build (fails) after that

sequence 2: catkin clean -y catkin build -v (works) catkin build (sometimes work, sometimes fails)

sequence 3: catkin clean -y catkin build -v (works) catkin build -v (works) after that

sequence 4: catkin clean -y catkin build (fails) catkin build -v (works) after that

One thing to note: These are the patterns that we experience. Once in a while the catkin build works that doesn't follow this pattern (like on the first try after a clean) although this is rare. I agree this is probably a race condition issue. I tried to debug the python script by executing it within a debugger. Except there seems to be a lot of parallel processes that make it hard to tell what is going on. Also, a lot of loops occur multiple times and it is hard to break near the time the error occurs.

[wjwwood]

Hmm, that is strange though. I would have thought a "use before generation" race condition would not cause repeated failures, unless the generated file is removed and re-generated each time. It might be a race somewhere else.

Unfortunately I don't have time to dig into this issue right now, but thanks for the detailed issue. Maybe someone else with the issue can also look at it and make some progress.

[tiusty]

Ok, thanks for your help! I will keep digging myself and see if I can find something.

[VictorLamoine]

I have the same issue;

$ catkin build -Dram_documentation_GENERATE=ON ram_documentation
------------------------------------------------------------------------------
Profile:                     default
Extending:             [env] /opt/ros/kinetic
Workspace:                   /home/victor/code/_catkin_tools_workspace
------------------------------------------------------------------------------
Source Space:       [exists] /home/victor/code/_catkin_tools_workspace/src
Log Space:         [missing] /home/victor/code/_catkin_tools_workspace/logs
Build Space:        [exists] /home/victor/code/_catkin_tools_workspace/build
Devel Space:        [exists] /home/victor/code/_catkin_tools_workspace/devel
Install Space:      [unused] /home/victor/code/_catkin_tools_workspace/install
DESTDIR:            [unused] None
------------------------------------------------------------------------------
Devel Space Layout:          linked
Install Space Layout:        None
------------------------------------------------------------------------------
Additional CMake Args:       -Dram_documentation_GENERATE=ON
Additional Make Args:        None
Additional catkin Make Args: None
Internal Make Job Server:    True
Cache Job Environments:      False
------------------------------------------------------------------------------
Whitelisted Packages:        None
Blacklisted Packages:        None
------------------------------------------------------------------------------
Workspace configuration appears valid.
------------------------------------------------------------------------------
[build] Found '9' packages in 0.0 seconds.                                                                                                                                                                  
[build] Updating package table.                                                                                                                                                                             
Warning: generated devel space setup files have been deleted.
Starting  >>> catkin_tools_prebuild                                                                                                                                                                         
Finished  <<< catkin_tools_prebuild                [ 1.2 seconds ]                                                                                                                                          
Starting  >>> ram_documentation                                                                                                                                                                             
No handlers could be found for logger "trollius"d] [ram_documentation:make (0%) - 2.5]                                                                                                                      
[build] An internal error occurred!                                                                                                                                                                         
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/catkin_tools/verbs/catkin_build/build.py", line 549, in build_isolated_workspace
    continue_without_deps=False))
  File "/usr/lib/python2.7/dist-packages/catkin_tools/execution/executor.py", line 370, in run_until_complete
    return loop.run_until_complete(coroutine)
  File "/usr/lib/python2.7/dist-packages/trollius/base_events.py", line 350, in run_until_complete
    return future.result()
  File "/usr/lib/python2.7/dist-packages/trollius/futures.py", line 285, in result
    compat.reraise(type(self._exception), self._exception, exc_tb)
  File "/usr/lib/python2.7/dist-packages/trollius/tasks.py", line 259, in _step
    result = coro.send(value)
  File "/usr/lib/python2.7/dist-packages/catkin_tools/execution/executor.py", line 165, in async_job
    stdout=logger.get_stdout_log(),
  File "/usr/lib/python2.7/dist-packages/catkin_tools/execution/io.py", line 115, in get_stdout_log
    return self._decode(self.stdout_buffer)
  File "/usr/lib/python2.7/dist-packages/catkin_tools/execution/io.py", line 131, in _decode
    return data.decode('utf-8')
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 9530: invalid continuation byte

Example package here: https://gitlab.com/InstitutMaupertuis/ros_additive_manufacturing/tree/melodic/ram_documentation

[robotsorcerer]

Has this been fixed? I am finding the same issue on melodic.

[tiusty]

Unfortunately, I haven't been able to look more into this yet.

[argenos]

I'm also getting this error, but catkin build -v does not seem to help:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/catkin_tools/execution/stages.py", line 174, in function_proxy
    return function(logger, event_queue, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/catkin_tools/jobs/catkin.py", line 291, in link_devel_products
    source_hash = md5(open(os.path.realpath(source_file)).read().encode('utf-8')).hexdigest()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd8 in position 40: ordinal not in range(128)

Stage `symlink` failed with arguments:
  package: {'buildtool_export_depends': [], 'test_depends': [<catkin_pkg.package.Dependency object at 0x7fd927fe9440>], 'exec_depends': [<catkin_pkg.package.Dependency object at 0x7fd927fe94b0>, <catkin_pkg.package.Dependency object at 0x7fd927fe9520>], 'build_depends': [<catkin_pkg.package.Dependency object at 0x7fd927fe9590>, <catkin_pkg.package.Dependency object at 0x7fd927fe9600>, <catkin_pkg.package.Dependency object at 0x7fd927fe9670>, <catkin_pkg.package.Dependency object at 0x7fd927fe96e0>, <catkin_pkg.package.Dependency object at 0x7fd927fe9750>, <catkin_pkg.package.Dependency object at 0x7fd927fe97c0>, <catkin_pkg.package.Dependency object at 0x7fd927fe9830>, <catkin_pkg.package.Dependency object at 0x7fd927fe98a0>, <catkin_pkg.package.Dependency object at 0x7fd927fe9910>, <catkin_pkg.package.Dependency object at 0x7fd927fe9980>], 'licenses': ['GPLv3'], 'group_depends': [], 'doc_depends': [], 'filename': '/home/argen/ros/mas_stable/src/mas_perception/mcr_scene_segmentation/package.xml', 'version': '0.0.1', 'package_format': 1, 'exports': [], 'description': u'Scene segmentation', 'build_export_depends': [<catkin_pkg.package.Dependency object at 0x7fd927fe99f0>, <catkin_pkg.package.Dependency object at 0x7fd927fe9a60>], 'maintainers': [<catkin_pkg.package.Person object at 0x7fd927feaa50>], 'member_of_groups': [], 'authors': [<catkin_pkg.package.Person object at 0x7fd927feaad0>, <catkin_pkg.package.Person object at 0x7fd927feab50>, <catkin_pkg.package.Person object at 0x7fd927feabd0>], 'conflicts': [], 'buildtool_depends': [<catkin_pkg.package.Dependency object at 0x7fd927fe9ad0>], 'version_compatibility': None, 'replaces': [], 'name': 'mcr_scene_segmentation', 'urls': []}
  metadata_path: /home/argen/ros/mas_stable/.catkin_tools/profiles/default
  devel_manifest_path: /home/argen/ros/mas_stable/.catkin_tools/profiles/default/packages/mcr_scene_segmentation
  prebuild: False
  package_path: mas_perception/mcr_scene_segmentation
  dest_devel_path: /home/argen/ros/mas_stable/devel
  source_devel_path: /home/argen/ros/mas_stable/devel/.private/mcr_scene_segmentation

[Crowdedlight]

Whats the status on this issue? And when does the fix make it into a release? :)

I am getting same issue but only on my CI catkin run_tests, locally it works. My CI also success when doing catkin build, but fails consistently on catkin run_tests

[rukie]

This is also an issue for me.

[ConnorCrwfNRG]

Ok, so I think I found a fix. What I did was:

1. remove python-catkin-tools with sudo apt when both system python environments are active (change with sudo update-alternatives --config python)
2. remove catkin_tools with pip uninstall catkin_tools when both system python environments are active This basically removes any references to catkin_tools in any /bin or /lib folders underneath various parent directories (in /home or /usr)
3. Verify that package can be built with catkin_make by running the following: 3a. rosdep install --from-paths src/ --ignore-src --rosdistro melodic 3b. catkin_make --only-pkg-with-deps
4. Assuming it built fine, rm -r the now created /build and /devel folders in catkin_ws (should have done a catkin clean --deinit at the beginning of all this before uninstalling catkin tools by the way)
5. Now re-install catkin tools with sudo pip install -U catkin_tools (don't be afraid of sudo pip!, it just installs to /usr/local/bin and /lib, no big deal, right?)
6. Re-initialize catkin_ws with catkin init and try to build troublesome package again (os symlink errors should be gone now and package should build fine).

Let me know if this works for y'all.

[Tuebel]

I had a similar issue when using a system library - assimp in my case. When trying to add it to the DEPENDS section catkin_package, I would get the same No handlers could be found for logger "trollius"ed] :

catkin_package(
  INCLUDE_DIRS include
  LIBRARIES scigl_render_ros
  CATKIN_DEPENDS 
    cv_bridge
    dynamic_reconfigure
    image_geometry
    image_transport
    roscpp
    tf2_geometry_msgs
    tf2_ros
  DEPENDS
    ASSIMP
    gl3w
    scigl_render)

Compiling the project via catkin_make_isolated gave me the hint with a different error message: Project 'scigl_render_ros' specifies '/usr/lib/include' as an include dir, which is not found.. Removing the Assimp from the the catkin_package and manually linking it in the downstream package solved the problem for both, catkin build and catkin_make_isolated.

[a7g4]

I just ran into this issue and did some digging to fix it locally. It turns out this is already fixed by #399.