Real time benchmarking / reasoning

[fmauch]

This PR aims at providing more detailed information about setting up a system for stable communication with the robot.

This is best combined with https://github.com/UniversalRobots/Universal_Robots_Client_Library/pull/139

Planned additions (maybe other PRs)"

• Support non-blocking read in our HW-Interface node
• Update PREEMPT_RT build instructions

[github-actions[bot]]

This PR hasn't made any progress for quite some time and will be closed soon. Please comment if it is still relevant.

[shuobh]

One additional point for comparison -- We've integrated the proper freedrive mode and compared the performance by switching between trajectory execution and freedrive mode for 20k trajectories. One with low latency kernel fails 1% of the time with error saying that it can not execute trajectory in freedrive mode yet real time kernel only failed once.

I also want to circle back to an issue about sched_FIFO. We tried to dive a bit into it and somehow it is related to user privilege, where setting priority returns "Operation not permitted". Some suggests to set the application to run with sudo privilege but it is not very easy for ROS and it might introduce other issues. Is there another way to fix this?

[fmauch]

I also want to circle back to an issue about sched_FIFO. We tried to dive a bit into it and somehow it is related to user privilege, where setting priority returns "Operation not permitted". Some suggests to set the application to run with sudo privilege but it is not very easy for ROS and it might introduce other issues. Is there another way to fix this?

We have that covered by our tutorial. You can give your user the privilege to set the priority. I would not recommend running the application using sudo.

[shuobh]

I also want to circle back to an issue about sched_FIFO. We tried to dive a bit into it and somehow it is related to user privilege, where setting priority returns "Operation not permitted". Some suggests to set the application to run with sudo privilege but it is not very easy for ROS and it might introduce other issues. Is there another way to fix this?

We have that covered by our tutorial. You can give your user the privilege to set the priority. I would not recommend running the application using sudo.

I followed the tutorial to set user privileges but the problem is still there, same as other people mentioned in the post I linked. I've pasted my setup below.

infinity@nuc-41-robot-35:~$ id -Gn
infinity adm cdrom sudo audio dip plugdev lpadmin pulse pulse-access lxd sambashare realtime

infinity@nuc-41-robot-35:~$ cat /etc/security/limits.conf
# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - a user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#        - NOTE: group and wildcard limits are not applied to root.
#          To apply a limit to the root user, <domain> must be
#          the literal username root.
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open file descriptors
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#        - chroot - change root to directory (Debian-specific)
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#root            hard    core            100000
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#ftp             -       chroot          /ftp
#@student        -       maxlogins       4
@realtime soft rtprio 99
@realtime soft priority 99
@realtime soft memlock 102400
@realtime hard rtprio 99
@realtime hard priority 99
@realtime hard memlock 102400
# End of file

[fmauch]

@shuobh tbh I don't know what's going wrong there. However, this bringing discussions in this PR out of line. Please open a separate issue for that. Although, as I said, I don't quite know what is going on there.

[samialperen]

@shuobh @fmauch What is the current status on this on ROS2 side? I think documentation is not super up to date. What is the suggested way to get the best performance? I see that in this MR a comment by @fmauch says Regardless of the actual kernel used, setting the producer thread to FIFO scheduling seems to improve robustness.. However on this thread, @shuobh suggesting that rt kernel performs better based on his tests. Could you guys shed a bit more light on this? The documentation on the repos are not up to date :(