IntelRealSense
So many issues on Raspberry Pi (4 B) (using ROS)
First, I am trying to use the D455 on a headless Pi 4B so I can't try any of the visualisation tools that come with the SDK. This is all on an up to date Ubuntu 22.04.4.
I have installed from sources librealsense-2.54.2 using FORCE_RSUSB_BACKEND, as recommended on so many reported issues here. I also had to disable BUILD_GLSL_EXTENSIONS as the build looks for glfw3 in the wrong location. After this, rs-enumerate-devices does detect my camera.
I then cloned the realsense-ros (ROS2) set of packages in my workspace and compiled them there.
I managed to get an RGB image (after enabling it) as well as a depth image. I can't get a pointcloud (even after enabling it).
However, everything is so flaky that it makes it very unusable. In particular, if I stop the ROS node, I have to disconnect the camera, start the node, wait for it to fail because the camera is not there, kill the node, reconnect the camera and restart the node. The it works again (still no pointcloud). The same applies to rs-enumerate-devices. I tried initial_reset:=true, but this just seems to hang and not do anything.
I seem to be getting many different messages, of the type:
[realsense2_camera_node-1] 11/03 16:02:13,923 WARNING [281471956805856] (messenger-libusb.cpp:42) control_transfer returned error, index: 300, error: Success, number: 0
or
[realsense2_camera_node-1] 11/03 16:12:33,503 WARNING [281472485222624] (messenger-libusb.cpp:42) control_transfer returned error, index: 300, error: Resource temporarily unavailable, number: b
or
[realsense2_camera_node-1] 11/03 16:13:33,106 WARNING [281472023914720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
With this last one I get depth images but no colour or pointcloud, although the topic are there. What I manage to get or not seemed very random. I never got a pointcloud.
Any idea what I can try?
Hi @flabrosse Are you using the official 1 meter USB cable supplied with the camera or another cable of your own choice, please? RealSense cameras require a high quality cable suited to data transfer, so if you use your own cable then that cable can negatively affect performance and can cause errors if it is not a high quality one (for example, an inexpensive cable suited only for device charging).
control_transfer returned messages, if they are generating continuously in the log, indicate that there is a communication problem between the camera and the computing device, such as an issue with the USB port or USB cable. Sometimes it is still possible to receive a normal depth and color stream in RViz whilst these messages are generating though and if that happens then it is safe to ignore the control_transfer messages.
Does the pointcloud appear if you launch with ros2 run instead of ros2 launch if you have not done so already?
ros2 run realsense2_camera realsense2_camera_node --ros-args -p pointcloud.enable:=true
Raspberry Pi boards can stream depth and color but are known to sometimes have problems with generating pointclouds in the RealSense ROS wrapper. In https://github.com/IntelRealSense/realsense-ros/issues/1967#issuecomment-1177569559 a RealSense ROS user with a Pi 4 achieved a pointcloud by using 640x480 resolution and 6 FPS. With the D455 model, 5 FPS would be used instead.
Thank you @MartyG-RealSense for your input. Some progress today, but still...
Using the following command, I got something to work more or less reliably, with RGB, depth and coloured pointcloud:
ros2 run realsense2_camera realsense2_camera_node --ros-args -p pointcloud.enable:=true -p depth_module.profile:=424x240x5 -p rgb_camera.profile:=424x240x5 -p enable_color:=true -p pointcloud.stream_filter:=2
I still get many error messages, such as:
12/03 16:16:07,569 WARNING [281472979822816] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
or
12/03 16:16:21,335 ERROR [281472065661152] (synthetic-stream.cpp:51) Exception was thrown during callback: Out of frame resources!
or
12/03 16:34:08,599 WARNING [281471956805856] (sensor.cpp:406) Frame received with streaming inactive,Depth0, Arrived,0.000000 1710261248599.315186
I tried 4 different USB cables, 2 different d455 cameras and different ports on the Raspberry Pi. All the same. I get lots of these "control_transfer" errors.
Using USB2 is extremely unreliable. This is annoying because USB3 creates all sorts of interference with the GNSS receiver that is next to the camera.
Any suggestions on all the above?
Apologies for the delay in responding further.
Do the streams render normally in RViz despite the warning messages in the terminal, as you mentioned that RGB, depth and coloured pointcloud were working more or less reliably when using a low resolution. If they do then then the messages can be safely ignored, even though they may be an inconvenience.
It is likely to be the pointcloud generation that is putting a processing burden on the Pi. Raspberry Pi boards can provide depth and color streams when used with a RealSense camera but may experience problems when required to do more than that, such as pointcloud generation or depth-color alignment.
The link below suggests various methods for dealing with interference from USB 3, such as using a shielded USB 3 cable or increasing the distance between the cable and the affected device, or using the affected device on a USB 2 connection if it is a USB device (which could be done by using a USB 2 cable or USB 2 hub with it if you do not have a USB 2 port).
https://www.rshtech.com/blog/how-to-avoid-the-usb30-and-24-ghz-devices-interference-2
Sorry, been busy with other things. I'll work on this again later today.
It seems that the realsense node starts better if I wait until others have started before trying it, so will work on this. I'll also see if I can get away with an RGBD image rather than a full point cloud, but something like rtab-map will have to produce a pointcloud anyway.
About USB2 vs USB3 and GPS noise, running the GPS on USB2 makes no difference (the noise is on the antenna side, not the USB). We cannot distance the cables and antenna as we are on a small robot (about 40cm long). What did help in the past was to run the realsense on USB2 rather than USB3 but this seems not to work anymore (but maybe doing RGBD instead of PC will help). More to play with.
I might also swap to a Pi 5.
No problem at all. Thanks very much for the update.
At the time of writing this we have only had a few Pi 5 cases but so far users have found that RealSense Pi 5 projects still have problems.
I look forward to your next update after tests. Good luck!
Hi @flabrosse Do you require further assistance with this case, please? Thanks!
Sorry, have not had time to work on this recently. Last time I tried, it was kind of working, in that eventually everything was fine, but needing many starting of the node, killing it, etc. I need to tru some more to see if I can reproduce this.
It's no problem at all. I look forward to your next report. Good luck!
Hi @flabrosse Do you have an update about this case that you can provide, please? Thanks!
Finally coming back to this, to find that the situation has not improved. For now I am failing to get anything from the camera, even without attempting to generate a pointcloud.
From a fresh start of the RPi, I seems to work (I get lots of control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11 for example though).
If I kill my node and try to restart it, it does not work. It seems to fail to connect to the camera and I eventually get control_transfer returned error, index: 300, error: Success, number: 0. rs-enumerate-devices also hangs and eventually fails.
If I then disconnect and reconnect the camera, rs-enumerate-devices does see the camera, the ros node manages to connect to the camera but I get no data and lots of control_transfer returned error, index: 300, error: Resource temporarily unavailable, number: b.
Not sure what else I can try...
If I start the node with the camera disconnected, the node fails saying there is no camera. I then kill the node, connect the camera, restart the node and it works, kind of. Sometimes one or the other of color or depth image is missing.
Are you attaching the camera directly to a USB port on the Pi or plugging it into a USB hub? The symptoms that you describe (reset not working but a shutdown-reboot does work) could be caused by a USB controller that is not fully compatible with the camera.
The operation of USB ports are handled by a USB controller, and the brand-name of controller that is used is chosen by the manufacturer (of the computer or of the USB hub). So a USB hub will likely have a different brand of USB controller to the brand used on the computer's ports because they are made by different manufacturers.
This means that if the camera does not work correctly when plugged directly into a computer's ports then there is a chance that it may work if plugged into a port on a USB hub because of the hub's different controller. And a camera that does not work on a particular hub might work when plugged into the computer's ports.
This is directly on one of the USB3 ports on the RPi, although ultimately I would prefer to run it on a USB2 port to reduce EM noise on the GPS.
The specification for Pi 4B lists it as having two USB 3 ports and two USB 2 ports. Raspberry Pi 5 has the same set of USB ports.
Yes, I confirm the RPi we have has 2 USB3 ports and 2 USB2 ports. I am not sure what the point of your remark was.
You mentioned that you would prefer to run it on a USB2 port so I was making sure that you were aware that Pi 4's ports were not all USB3.
I see your point. Yes, I am aware that there are also USB2 ports. However, at the moment the camera is not reliable enough using its native USB3 mode, I am expecting it to be worse on USB2, but maybe not. I will try with a hub, but this is not really an option in our case because of very limited space.
The main effects of using USB2 with a RealSense camera on resolutions higher than 640x480 are that you will be limited to using the D455's minimum FPS speed of '5', and the data transfer speed of USB2 is slower (480 megabits per seocnd / Mbps for USB2 and 5000 Mbps for USB3).
With 640x480 resolution, 30 FPS is supported on USB2.
OK. I'm using the lowest resolution at the lowest FPS anyway. The RPi is not very fast...
Tried to connect the camera via 2 different hubs, one ANKER USB3, one BELKIN USB2. Every time rs-enumerate-devices gives me:
28/05 14:16:37,125 ERROR [281473770647776] (handle-libusb.h:53) failed to open usb interface: 0, error: RS2_USB_STATUS_ACCESS
28/05 14:16:37,126 ERROR [281473811937888] (sensor.cpp:661) acquire_power failed: failed to set power state
Could not create device - failed to set power state . Check SDK logs for details```
I have also tried on a USB2 port directly on the RPI and the streaming of data (only colour and depth at the lowest resolution and fps) is very unreliable, going OK then stopping, then going again, etc.
Are either of the hubs that you tried powered by a mains electricity plug socket (i.e they draw thei power from the mains instead of the Pi board), or is a mains power connection not practical on your particular project?
Small mobile robot, so definitely mot possible. However, we checked the 5V line and it never went under 5V.
A RealSense user of Pi shared a Python script at https://github.com/IntelRealSense/librealsense/issues/9022#issuecomment-840975425 that improved performance by creating separate frame queues for the depth and color streams. That approach will not work for you though as you are using ROS.
Does setting the Create visualization settings of RViz to use compressed data instead of raw data - as described at https://github.com/IntelRealSense/realsense-ros/issues/1903#issuecomment-898522352 - make a difference to performance?
I guess the ROS node could use the approach of the code linked (although I am not convinced there was really a consensus on whether the code does the correct thing or not).
To recap, I have two different issues.
(1) Sometimes the topics are not created, sometimes they are but nothing is published on them. Usually, but not always, disconnecting the camera and reconnecting it helps. Trying to start the software with the camera disconnected, making it realise there is no camera, stopping the software, reconnecting the camera and restarting the software seems to make it succeed more often, but is not systematic.
(2) USB3 is more reliable then USB2. Even at the lowest resolution and fps USB2 is erratic, with tens of seconds not getting any new frames, then getting some, etc. I'll see if using compressed streams helps, and if there are any system logs during these times.
Earlier in this discussion I suggested using a shielded USB 3 cable to help to avoid interference with your GNSS receiver. As you are using a small robot, you could research short shielded USB 3 cables of 0.25m or 0.5m length using the search term 'shielded usb3 cable 0.25m' or 'shielded usb3 cable 0.5m'
We have tried on other robots to add a shield to cables, using a steel braid rolled around the cable, with not much success. I will look for other cables, but I am not holding my breath.
Hi @flabrosse Were you able to find other cables to try, please? Thanks!
the moment I am still at the stage of trying to make the camera work reliably.
The latest hurdle: I installed rtabmap which requires the system realsense packages that don't work on the RPi. After a rebuild of my packages, nothing works because the system packages are picked up. Need to figure out how to install rtabmap without the realsense stuff or how to make my packages pickup the right libraries (not just the ros stuff but also the underlying librealsense).