Session daemon error during Caret recording in Autoware

[ThankPan]

Checklist

• [X] I've read the contribution guidelines.
• [X] I've read document and no solution were found.

Issue type

Bug

Step where the issue is related to

Recording

Environments

• OS: Ubuntu 22.04.3 LTS
• Kernel version: 5.15.0-91-generic
• ROS 2: Humble
• CPU: Intel® Core™ i7-9700K CPU @ 3.60GHz × 8
• Memory: 32.0 GiB

CARET version

• CARET: v0.4.22

Target application

• Application: Autoware running with AWSIM.

Description

I have encountered Error: No session daemon is available. or recvmsg: Connection reset by peer (in lttcomm_recv_unix_sock() at unix.c:205) Error: Fatal error of the session daemon. I think the error is caused by lttsession failure. This causes limited nodes and topics trace was recorded. I reckon it is caused by too many caret_trace nodes created and leading to some error in lttsession. I used the launch file from the carete_autoware_launch repository and I believe the session was correctedly created since I could collect certain nodes' traces. However, it is not complete and a lot of nodes are missing when I analyze the records. Any possible solution to this?

Steps to reproduce

1. Install caret.
2. Compile autoware with caret.
3. Modify launch files.
4. Start AWSIM.
5. Launch Autoware.

Commands

No response

Logs

No response

LTTng trace log file

No response

Screenshots

No response

[ymski]

I apologize for the delayed response @ThankPan .

I haven't encountered the errors recvmsg: Connection reset by peer (in lttcomm_recv_unix_sock() at unix.c:205) Error: Fatal error of the session daemon. Therefore, it is challenging to provide specific solutions at this moment. If we gain any insights, we will contact you.

[xygyo77]

@ThankPan Thank you for your patience. The caret_autoware_launch you are trying does not support AWSIM. Please try again using the following procedure.

environments

# installed ubuntu22.04 and ros2 humble-desktop
ulimit -n 16384

caret setup

source /opt/ros/humble/setup.bash
cd ~
git clone [email protected]:tier4/caret.git
cd ros_caret_ws
mkdir src
vcs import src < caret.repos
./setup_caret.sh
colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release

autoware setup

source /opt/ros/humble/setup.bash
cd ~
git clone https://github.com/autowarefoundation/autoware.git
cd autoware
git checkout 2024.03

# If the cuda/cudnn/tensorrt environment has been established, the cuda setup will skip. 
# if you do not want to install only nvidia drivers, submit the command
# ./setup-dev-env.sh -y --no-cuda-drivers --download-artifacts

./setup-dev-env.sh

nvidia-smi # Ensure success

mkdir src
vcs import src < autoware.repos

rosdep update
rosdep install -y --from-paths src --ignore-src --rosdistro $ROS_DISTRO

https://tier4.github.io/caret_doc/main/faq/known_issues/#build Apply the patch indicated in this known_issues.

# build
source ~/caret_ws/ros2_caret_ws/install/local_setup.bash

colcon build --symlink-install --packages-skip rclcpp rclcpp_action rclcpp_components rclcpp_lifecycle --cmake-args -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=Off -DCMAKE_CXX_FLAGS="-w"

# Confirm that caret has been successfully applied and built
. ~/ros2_caret_ws/install/local_setup.bash
xxx@yyy:~/autoware$ ros2 caret check_caret_rclcpp .
INFO    : 2024-07-23 10:49:20 | All packages are built using caret-rclcpp.

AWSIM setup

https://tier4.github.io/AWSIM/GettingStarted/QuickStartDemo/ Get AWSIM and map data. autoware has already been built and is not needed her

---

AWSIM caret measurement.

terminal#1 AWSIM

cd ~
AWSIM/AWSIM_v1.2.2/AWSIM.x86_64

terminal#2 autoware logging_simulator Once rviz starts, wait for Localzation to become Intialized and set the goal.

source /opt/ros/humble/setup.bash
cd ~/autoware
source ~/ros2_caret_ws/setenv_caret.bash
source  install/local_setup.bash
ros2 launch autoware_launch e2e_simulator.launch.xml vehicle_model:=sample_vehicle sensor_model:=awsim_sensor_kit map_path:=<your mapfile location>

terminal#3 caret record

source /opt/ros/humble/setup.bash
source ~/ros2_caret_ws/install/local_setup.bash
ros2 caret record -v -s trace_data -f 50000 --light
UST tracing enabled (xx events)
kernel tracing disabled
context (x fields)
writing tracing session to: /home/xxx/.ros/tracing/trace_data
press enter to start...

The first <ENTER> input starts the measurement, and the second <ENTER> input ends the measurement. Measurement data is generated under /home/xxx/.ros/tracing/trace_data.

[xygyo77]

@ThankPan Since we have not received a response for a long time and the status of the issue is unknown, we are closing it. If you encounter similar problems in the future using the method shown here, please create another issue.