generating compile_commands.json for workspace

[rhklite]

System Info

• Operating System:

Linux thousandsunny 4.15.0-50-generic #54~16.04.1-Ubuntu SMP Wed May 8 15:55:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• Python Version: Python 2.7.12
• Version of catkin_tools:

catkin_tools 0.4.5 (C) 2014-2019 Open Source Robotics Foundation
catkin_tools is released under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)
---
Using Python 2.7.12 (default, Nov 12 2018, 14:36:49) [GCC 5.4.0 20160609]

• Git version: git version 2.7.4
• ROS Distro: kinetic

Build / Run Issue

• [x] Works with catkin_make
• [ ] Works with catkin_make_isolated --merge
• [ ] Works with catkin build
• [ ] Works with catkin build -p1
• [ ] I did not read this

Expected Behavior

• catkin build -DCMAKE_EXPORT_COMPILE_COMMANDS=1 should generate a compile_commands.json file inside catkin_ws/build that's for the entire workspace. I'm using this file for the cquery linter

Actual Behavior

• I'm trying to get a compile_commands.json output with catkin build, the same file that you would get when you use cakin_make
• catkin_make: catkin_make -DCMAKE_EXPORT_COMPILE_COMMANDS=1 would generate a compile_commands.json inside catkin_ws/build which is a nice file that works for the entire workspace.
• catkin build: catkin build -DCMAKE_EXPORT_COMPILE_COMMANDS=1 doesn't generate a nice file for the entire workspace, but rather individual compile_commands.json for each package. I tried to do a work around by using a python file to concatenate all the compile_commands.json in each subfolder in the build folder. However the generated compile_commands.json doesn't recognise #include <message_header>.h, where as the compile_commands.json generated through catkin_make does.

Is there a way to properly generate a compile_commands.json for the entire catkin workspace using catkin build?

python script here:

import os
import sys

workspace_path = os.popen('catkin locate --workspace $(pwd)').read().rstrip()

if not workspace_path:
    sys.exit("\033[1;31mNo workspace found in the current directory\033[0m")


ccjson_path = workspace_path+'/compile_commands.json'
target_ccjson_path = workspace_path+'/src'+'/compile_commands.json'

for i in [ccjson_path, target_ccjson_path]:
    if os.path.isfile(i):
        os.system('rm %s' % i)
        print("removed previous compile_commands.json located at %s" % i)

# create a dictionary with file names as keys
# and for each file name the paths where they
# were found
file_paths = {}
for root, dirs, files in os.walk(workspace_path):
    for f in files:
        if f.endswith('compile_commands.json'):
            if f not in file_paths:
                file_paths[f] = []
            file_paths[f].append(root)

# for each file in the dictionary, concatenate
# the content of the files in each directory
# and write the merged content into a file
# with the same name at the top directory
for f, paths in file_paths.items():
    txt = []
    with open(os.path.join(paths[0], f)) as f2:
        txt.append('['+f2.read()[1:-2]+',')
    for p in paths[1:-1]:
        with open(os.path.join(p, f)) as f2:
            txt.append(f2.read()[1:-2]+',')
    with open(os.path.join(paths[-1], f)) as f2:
        txt.append(f2.read()[1:-1]+']')
    with open(f, 'w') as f3:
        f3.write(''.join(txt))

if not os.path.isfile(target_ccjson_path):
    os.system('mv ./compile_commands.json %s' % (target_ccjson_path))


print("\033[1;32mFinished writing compile_commands.json\033[0m")

Steps to Reproduce the Issue

cd ~/catkin_ws
catkin clean
catkin build -DCMAKE_EXPORT_COMPILE_COMMANDS=1

Edit: Made the python script work anywhere in the catkin workspace

[rhklite]

there was a bug in my work around python script, i made edits to the script and got it working.

Is there a better way to do this? to generate a compile_commands.json without needing a pythonc script?

[ndepal]

I agree, it would be great if catkin build -DCMAKE_EXPORT_COMPILE_COMMANDS=1 did this.

Based on your python script I made my own shell script that does the same:

#!/bin/sh

cd `catkin locate --workspace $(pwd)`

concatenated="build/compile_commands.json"

echo "[" > $concatenated

first=1
for d in build/*
do
    f="$d/compile_commands.json"

    if test -f "$f"; then
        if [ $first -eq 0 ]; then
            echo "," >> $concatenated
        fi

        cat $f | sed '1d;$d' >> $concatenated
    fi

    first=0
done

echo "]" >> $concatenated

[kgreenek]

If you happen to use YouCompleteMe with vim, you can use this plugin so you don't have to generate the merged compile_commands.json file. Just run catkin config --cmake-args='-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', install the plugin to vim, and everything will just work when you're inside the catkin workspace. Alternatively, you can place the .ycm_extra_conf.py file form that repo in the root of your catkin workspace.

https://github.com/kgreenek/vim-ros-ycm

[LeroyR]

It generates multiple compile commands instead of one merged file as you are working with multiple projects inside. Merging the compile commands sounds like breaking the separation between multiple projects.

So while i understand why you/we/i would want this, it does not sound like something catkin tools should ever do.

edit: catkin_make works since it mushes everything together resulting in packages that seem to work but break if used without the other parts in your workspace (or if build with catkin tools).

[AlexisTM]

@ndepal @rhklite

My way to merge all compile_commands.json together, using the jq tool to simplify integration with other tools.

sudo apt install jq

jq -s 'map(.[])' build_folder/**/compile_commands.json > compile_commands.json

@LeroyR I agree, catkin_tools should keep compile commands split as it is a split workflow.

[hgaiser]

I am doing something similar. It isn't pretty, but it doesn't require an extra tool:

printf '[' > compile_commands.json
find ./build_isolated -type f -name 'compile_commands.json' -exec sh -c "cat {} | tail -n+2 | head -n-1 && printf ','" >> compile_commands.json \;
sed -i '$s/.$//' compile_commands.json
printf '\n]\n' >> compile_commands.json

[stertingen]

Interestingly, colcon does this: https://github.com/colcon/colcon-cmake/pull/69

[zoenglinghou]

I used @AlexisTM 's comment and come up with a script

[wafts95]

I came across another use case for having a common compile_commands.json file, and that is to enable the use of the SciTools Understand IDE (link) with ROS projects.