ccache environment variable issue

[sasakisasaki]

Checklist

• [X] I've read the contribution guidelines.
• [X] I've searched other issues and no duplicate issues were found.
• [X] I'm convinced that this is not my fault but a bug.

Description

Hello! I'm a newcomer as the developer. I'm being so excited to work on this project! Just last week, I have started working with using this great work Autoware.

Maybe following my message contains something missing as I'm a beginner. I'll be happy if there'll be any feedback/comments!

As I saw the following issue during the compile time, please let me ask a question. Thank you so much in advance!!

• Question

  • Is it possible to fix following issue? :pray:

• Issue:

  • When compiling by colcon, I met the following error

---
Failed   <<< polar_grid [0.56s, exited with code 1]
--- output: time_utils
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /usr/lib/ccache/gcc
-- Check for working C compiler: /usr/lib/ccache/gcc - broken
CMake Error at /usr/share/cmake-3.22/Modules/CMakeTestCCompiler.cmake:69 (message):
  The C compiler

    "/usr/lib/ccache/gcc"

  is not able to compile a simple test program.

• Observation:
  • This error disappeared after running following commands:

export CCACHE_DIR=
export CC=
export CXX=

Related PR

• https://github.com/autowarefoundation/autoware/pull/4530

Expected behavior

• Build succeeds

Actual behavior

• Build did not work as the error message above

Steps to reproduce

$ git clone https://github.com/autowarefoundation/autoware.git
$ cd autoware
$ ./setup-dev-env.sh -y
$ mkdir src
$ vcs import src < autoware.repos
$ source /opt/ros/humble/setup.bash
$ source ~/.bashrc
$ rosdep install -y --from-paths src --ignore-src --rosdistro $ROS_DISTRO
$ colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release

Versions

• OS: Ubuntu 22.04.4 LTS
• ROS 2
• Commit Hash on this repo: 918a067f2345fc726ca9de5918860281c4ff1fb8

Possible causes

• Perhaps these 3 lines might be related

Additional context

No response

[shmpwk]

@oguzkaganozt cc. @xmfcx Could you check this issue...? I think this problem came from https://github.com/autowarefoundation/autoware/pull/4556.

[HansRobo]

@sasakisasaki Could you please try the following and see if it fixes it?

export PATH="/usr/lib/ccache/:$PATH"

related: https://github.com/autowarefoundation/autoware/pull/4530

[sasakisasaki]

@HansRobo Thank you so much! I tried the provided commands and investigated what is the possible fix.

I have a proposal for possible fix. I hope my idea will help to find the best idea.

• My Proposal For now, the .bashrc is modified so following three lines are added by ./setup-dev-env.sh -y

export CCACHE_DIR="/var/tmp/ccache"
export CC="/usr/lib/ccache/gcc"
export CXX="/usr/lib/ccache/g++"

(EDDITED) Perhaps, adding following lines might be able to handle more cases (my proposal):

# Set PATH (proposed by HansRobo san)
export PATH="/usr/lib/ccache/:$PATH"

# Set /var/tmp/ccache if exists otherwise set nothing
export CCACHE_DIR=$( [ -f /var/tmp/ccache ] && echo "/var/tmp/ccache" || echo "" )

# Set /usr/lib/ccache/gcc if exists otherwise set the output of `which gcc`
export CC=$( [ -f /usr/lib/ccache/gcc ] && echo "/usr/lib/ccache/gcc" || which gcc )

# Set /usr/lib/ccache/g++ if exists otherwise set the output of `which g++`
export CXX=$( [ -f /usr/lib/ccache/g++ ] && echo "/usr/lib/ccache/g++" || which g++ )

Although adding these lines worked on my environment, maybe this affects in case of bare-metal (your check is appreciated :pray: ). I think there are more choices for the solution. I'll be very happy if other ideas are shared!

• Reason of my Proposal After running following commands,

(under autoware folder)
export PATH="/usr/lib/ccache/:$PATH"
rm -rf build
colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release

I observed following error:

--- stderr: ament_cmake_gen_version_h                                                                                                                     
CMake Error at /usr/share/cmake-3.22/Modules/CMakeDetermineCCompiler.cmake:49 (message):
  Could not find compiler set in environment variable CC:

  /usr/lib/ccache/gcc.
Call Stack (most recent call first):
  CMakeLists.txt:2 (project)


CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
---

Thus, I guessed it might be needed to handle the case when the /usr/lib/ccache/gcc exists or not (same for $CXX and $CCACHE_DIR).

• Finally If needed, I'll create a merge request after understanding what is the better way for applying the fix without any side effect. So, any comments and feedbacks are highly appreciated!! Thank you very much in advance!!

[shmpwk]

@sasakisasaki Thank you for your suggestion. It looks great idea to avoid reading files that don't exist. I suggest it would be simpler to branch the condition by the presence or absence of ccache files before writing in the bashrc here. Currently ansible writes to bashrc in any case, but if it would not write out if the files don't exist, it would not pollute the bashrc.

[HansRobo]

@sasakisasaki @shmpwk

#4530 assumes that ccache is installed using the same setup script, so the binary presence check may have been omitted. However, it is quite possible that ccache on your development machine cannot be installed for some reason, or is deleted or no longer used after installation. So I think your suggestions are good! (From what I can see, I don't think there are any side effects from this proposal.)

If needed, I'll create a merge request after understanding what is the better way for applying the fix without any side effect.

There is also an option to suggest corrections to #4530 instead of creating new PR.

LINK: GitHub suggesting change feature

[sasakisasaki]

Thank you @HansRobo san for your comment and feedback! I added my suggestion onto the #4530 :+1:

[xmfcx]

@sasakisasaki @HansRobo @mebasoglu @oguzkaganozt

I don't understand why we need to set ccache in path.

On my local machine, If I follow these instructions: https://autowarefoundation.github.io/autoware-documentation/main/how-to-guides/others/advanced-usage-of-colcon/#using-ccache-to-speed-up-recompilation

which installs the ccache with sudo apt install ccache and set the env vars as:

export CC="/usr/lib/ccache/gcc"
export CXX="/usr/lib/ccache/g++"
export CCACHE_DIR="$HOME/.cache/ccache/"

Then everything works fine and I can check that all works by ccache -s.

So, apt installation should put it in its right place anyways. Why did you need to redefine it?

Could you install ccache with apt and try it with just these settings again?

I tried the ansible script on a fresh ubuntu installation before and it shouldn't create an issue with compilation.

[sasakisasaki]

@xmfcx Thank you so much for providing the detailed information!

Yes, you are right. After following the procedure written in the provided link, I can see the /usr/lib/ccache/gcc and /usr/lib/ccache/g++ are created. I needed to investigate/explore the documentation to understand why ccache is needed. Again, thank you for your kind feedback!

Sorry for making a confusion. Please let me explain the flow why I observed the missing ccache. I hope this will make the current situation clear.

Flow

• As my first study, I found this Source Installation
  • (At this stage, I had to investigate more documentations to know the possible installation patterns)
• Followed procedures = commands written in the "Source Installation"

sudo apt-get -y update
sudo apt-get -y install git
git clone https://github.com/autowarefoundation/autoware.git
cd autoware
./setup-dev-env.sh
mkdir src
vcs import src < autoware.repos
source /opt/ros/humble/setup.bash
rosdep install -y --from-paths src --ignore-src --rosdistro $ROS_DISTRO
colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release

• Found the /usr/lib/ccache/gcc and /usr/lib/ccache/g++ are missing in my environment at compile time
• Created this issue

According to setup-dev-env.sh, there is no ccache installation. Perhaps, sudo apt-get install -y ccache is needed in the setup-dev-env.sh (or maybe the documentation "Source Installation" needs a fix?). At this stage, what looks the better action item here?

Sorry if I'm saying something confusing again. I'll be happy for knowing/understanding your feedback. Thank you in advance!!

[xmfcx]

Thanks @sasakisasaki for providing detailed bug report💖, it's not your fault, we will investigate🩺.

https://github.com/autowarefoundation/autoware/blob/fb5115986cf2f4c42550560bfb4728e004a96804/setup-dev-env.sh#L79

https://github.com/autowarefoundation/autoware/blob/fb5115986cf2f4c42550560bfb4728e004a96804/ansible/playbooks/universe.yaml#L42

https://github.com/autowarefoundation/autoware/blob/fb5115986cf2f4c42550560bfb4728e004a96804/ansible/roles/build_tools/tasks/main.yaml#L1-L6

Following these steps, the setup-dev-env.sh should be able to install the latest ccache and configure it correctly.

I don't know why it didn't work for you but I will install on a fresh ubuntu again to see what happens and report here.

[sasakisasaki]

Thank you @xmfcx for kindly showing how the ccache is installed.

After understanding your feedback, I tried to replicate the situation why the /usr/lib/ccache is missing even after runinng ./setup-dev-env.sh. Then I found the issue looks related to my environment. My conclusion is, "the /usr/lib/ccache was deleted accidentally not by sudo apt purge -y ccache". That looks therefore my fault. I'm really sorry.

Reasons

• After seeing your first comment, I tried following commands:

sudo apt purge -y ccache
sudo apt install -y ccache

Then I could saw /usr/lib/ccache is created.

• Before that, I was trying following procedure multiple times

sudo apt-get -y update
sudo apt-get -y install git
git clone https://github.com/autowarefoundation/autoware.git
cd autoware
./setup-dev-env.sh
mkdir src
vcs import src < autoware.repos
source /opt/ros/humble/setup.bash
rosdep install -y --from-paths src --ignore-src --rosdistro $ROS_DISTRO
colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release

This multiple trial did not create the /usr/lib/ccache.

• So I considered like: "/usr/lib/ccache was not created by setup-dev-env.sh".
• But after trying following commands, the /usr/lib/ccache was created

sudo apt purge -y ccache
sudo apt install -y ccache

• Thus, my misunderstanding is derived by removing the /usr/lib/ccache not by sudo apt purge -y ccache which totally cleans up the ccache package correctly.
• I guess I wrongly removed the ccache during debugging

I did double-check if the ccache is installed after running:

sudo apt purge -y ccache
./setup-dev-env.sh -y

So the root cause of missing cache is should be due to my fault :bow: . Again, I'm so sorry for making the additional confusion. Now I understood I must use the totally clean environment from next time for verification.

Thank you very much for your kind feedback. From next time, I'll also check all the related scripts not only the shell script and also the ansible side too.

[sasakisasaki]

@xmfcx (CC: @shmpwk ) I found my proposal in this message is not needed as following reasons.

• We did not observe the missing ccache when doing clean installation multiple times
• My proposal is maybe dangerous:
  • The another compiler (not that of /usr/lib/ccache) might be used for ALL the Autoware's modules: this might lead unexpected result. Although the /usr/lib/ccache's compiler will be used unless unexpected operation, I guess it is maybe better to show an error when the /usr/lib/ccache does not exist rather than using compilers specified by which gcc and which g++.

This time, I learnt that being aware of the impact range by my fix is very important in the large-scale project.

Again, thank you for your feedback. I'll close this issue when all the remaining discussions are done.

[xmfcx]

So, for now we won't do any changes and all the PRs related to this discussion are either merged or closed. I'm closing this thread upon your comments.

Thanks for bringing this to our attention!