Running SLAM system on AirSim

[xuhao1]

Hi Guys, I have successfully run a well-known visual-inertial system VINS(https://github.com/HKUST-Aerial-Robotics/VINS-Fusion) on AirSim (which is amazingly good!) based on the ros package which verified the possibility of simulating my own aerial swarm in Airsim.

However, I found a few issues about the ros package;

1. The Unit of the angular velocity is wrong. The data from airsim is already rads
2. The coordinate system seems is confused
3. The unnecessary process lock lower the whole performance, with the useless lock we can only get IMU for ~20hz(because IMU always wait for images....For slam system we need high frequency IMU and low frequency images and depth), without we can get 300hz!
4. The ros package used simGetImages with single request a few times, which will be slow when we use a stereo camera. Change simGetImages with multiple requests will be much faster on mutliple cameras!
5. The ros package use soft time stamp (the receive ros::Time::now) instead simulation time
6. Nowhere to set host IP

For problem 1. 5. 6. I have submit a pull request #2368, For problem 234, my modification loses the generic but it solves the problem, gives me 20HZ stereo images and 300hz IMU on a remote notebook and successfully running VINS-Fusion in AirSim smoothly, here is my repo https://github.com/xuhao1/airsim_ros_pkgs. If you are interested in running VINS or other slam methods, you can have a try. The configs file for VINS and settings.json for Airsim have been placed in vins_airsim/ folder

[simengangstad]

We've worked some with VINS in AirSim as well, but haven't gotten any good results. Do you have any numbers on how much the position drifts compared to ground truth from your current setup?

[xuhao1]

@simengangstad , there is drift. However, drift is normal for VINS-Fusion without Loop Closure on. If you are interested in it, you can have a try with my package.

[Beniko95J]

Hi, may I ask how did you achieve 20Hz stereo images? I am also trying to run VINS-Mono on AirSim, and when I set the image resolution to 288x512 (original 144x256) and set the timer to save images every 0.05 second, I can only obtain 3Hz 288x512 images through its image apis. Do you have any idea about this?

[kminoda]

I am having similar issue in similar situation, which is to run VIO algorithm on AirSim ros pkg. I want to change frame rate of camera and IMU.

I assume that its "update_airsim_img_response_every_n_sec" in here that corresponds to camera frame rate. But it doesn't make any difference. Camera frame rate is still 3Hz even though I set the value to 0.1 (so it should be 10Hz).

I modified airsim_ros_wrapper.cpp according to this repos, but it doesn't solve the problem either.

Does anyone know how to solve this problem?

[dssdyx]

@xuhao1 Which coordinate did you transform imu link to? NED to ENU or other? I see your body_T_cam in vins yaml, is corresponding to ENU. The linear vel from NED to ENU should be (x,y,z)->(x,-y,-z),but your code is (x,y,z)->(-x,y,-z). I feel really confused.

[manthan99]

I am also having a similar issue with the published rate. I tried modifying the code according to this repo but I am only getting a publish rate of around 5 Hz for the camera and 10 Hz from IMU. Changing the "update_airsim_img_respone_every_n_sec" doesn't seem to increase the rate. Any solution to increase the rate?

I am having i7-6700HQ, 16 Gb ram and Gtx 1070

[rajat2004]

Could you check the parameter DrawDebugPoints in Lidar in the settings.json? If true, that causes a huge slowdown in the current latest binaries. Fix #2614 has been merged into master, but the problem is there in the binaries. Just change that to false and should improve the rates There's also another WIP PR #2472 to increase the camera rate

[manthan99]

@rajat2004 I am not using a Lidar. Here is the settings file which I am using [Settings.json]

Though the rate significantly improved to 20 Hz upon using the "NoDisplay" viewmode

[rajat2004]

Mentioned about the Lidar since that was causing problems with people trying out the ROS package tutorial and all. Using NoDisplay will definitely improve the rate since rendering is less @xuhao1 It would be great to have a PR with your changes if you're interested

[manthan99]

There seems to be a problem with the sync of images from left and right cameras ( ~0.05 sec) @xuhao1 did you face this issue?

[kminoda]

@manthan99 It’s nice to hear that you managed to get images on 20 Hz by using NoDisplay mode. Did you just change “ViewMode” to NoDisplay in the settings.json you posted?

[manthan99]

@kminoda Yes I just changed the "ViewMode" to NoDisplay in the settings.json file. In addition, I also replaced the airsim_node.cpp file that is given in this repo though I do not know if that made a difference. But still there is a problem with the syncing of both the images. The difference is pretty high (~0.05s)

[kminoda]

With an advice from a researcher who previously worked on running VIO in AirSim, I managed to run VINS-Mono on AirSim by changing "ClockSpeed" in settings.json. For me, since I could get 3Hz at maximum, I made ClockSpeed to 0.033 so that I can get 100Hz (in simulation time, not in actual time) for IMU. OFC it take much longer time to collect data even for short ones, but I can now estimate the pose of UAV without any drift.

[manthan99]

@kminoda Did you change the coordinate system for IMU ? And which parameters file did you use ?

[kminoda]

@manthan99 I didn’t change IMU frame (but maybe it’s better to do so). I verified that VINS-Mono is working by comparing the odometry output from vins and odom_local_ned. I changed the frame of odom_local_ned in comparing them. For parameters file (you mean the config for VINS-Mono/ Fusion, right?), I used this config file

[GimpelZhang]

@kminoda Yes I just changed the "ViewMode" to NoDisplay in the settings.json file. In addition, I also replaced the airsim_node.cpp file that is given in this repo though I do not know if that made a difference. But still there is a problem with the syncing of both the images. The difference is pretty high (~0.05s)

I wonder if you are still working on it.

The link posted by @xuhao1 : https://github.com/xuhao1/airsim_ros_pkgs may be able to solve the timestamps difference problem between the left and right images.

Main changes are in airsim_ros_wrapper.cpp and airsim_ros_wrapper.h. You may need to comment dji_sdk related parts which xuhao1 was using to connect with some particular hardwares.

Here is my modified version that might be just for reference. https://github.com/GimpelZhang/airsim_ros/tree/slam

The timestamps between left and right has a max ~0.0003s difference, which I think is good enough for stereo vslam algorithms. (I set the clockspeed to 0.1 when I tried)

P.S. I only checked the timestamps. I haven't used the data on any vslam algorithms yet.

[kminoda]

I also made it work by collecting IMU data with ROS while using computer vision mode for camera images (This way you can get completely synchronized images).

Here is a procedure.

• Use airsim_ros_pkg to record IMU and ground truth odometry msgs to generate a rosbag.
• Set AirSim to computer vision mode.
• Capture images from the point which you can get from gt odometry in the rosbag, and write these into it as rosmsg.

Hope it will help someone :)

[rajat2004]

If anyone wants to try the Faster Image Capture PR, they can try the binaries in https://github.com/microsoft/AirSim/pull/2713, On Linux need to run with -opengl Also any PRs to fix problems in ROS wrapper would be great :)

[dvdmc]

Since airsim_ros_wrapper changed quite a lot since this post, the referenced repos are old for being used as reference. I have added a timer to publish the IMU at a larger frequency in the main thread (as opposed to images and Lidar) link. I don't know if this is a good solution but still I'm not able to use VINS-Fusion. I think that now the problem is related to synchronization or the movement I'm using. Main complaints from VINS:

[ INFO] [1624356525.747085129]: misalign visual structure with IMU [ INFO] [1624356525.863072756]: Not enough features or parallax; Move device around

I am still having the problems of stereo pair synchronization and it is not clear for me if the coordinate system confusion is fixed in later versions of AirSim. Maybe the users who previously answered could provide updated answers or hints about the specific changes in the ROS wrapper code. Also, is there any intention of considering this issue in future versions of the official repo? I am willing to contribute :)

[phil333]

@davdmc That sounds like you having trouble configuring Vins-fusion? i needs a correct transformation matrix between the imu and each camera. If thats not properly set. Secondly, what environment are you using? is empty or are there any things that can be tracked I stopped going down this road because the left and right image are not synchronized. As long as this is not sorted, testing VO or VIO objectively is difficult, especially when producing larger images like for example the ZED camera. Essentially the images should be captured in parallel, but this might require freezing the sim, capturing and then unfreezing (not an expert on this).

[xxEoD2242]

@davdmc

I'll give you some general tips on how to utilize this:

1. You need to rewrite the image request structure of the ROS wrapper. Currently, the ROS wrapper is not written efficiently. I would recommend building your own thread that only gets the stereo images as a pair in the image request.
2. You need to increase the tolerance of VINS Fusion. The value is currently 0.03 but you need to increase that to 0.07.
3. Most likely your timestamps are not working properly for the images. I would look at those stamps and make sure they are in ROS time.

Just general pointers for working with VIO systems in AirSim. Happy to help with more specific issues.

[phil333]

@xxEoD2242 when you say tolerance, which value in vins fusion are you talking about? https://github.com/HKUST-Aerial-Robotics/VINS-Fusion/blob/master/config/euroc/euroc_stereo_imu_config.yaml

[GimpelZhang]

@xxEoD2242 when you say tolerance, which value in vins fusion are you talking about? https://github.com/HKUST-Aerial-Robotics/VINS-Fusion/blob/master/config/euroc/euroc_stereo_imu_config.yaml

I think @xxEoD2242 means the sync tolerance in the code processing stereo images: rosNodeTest.cpp

[phil333]

ok. There is a good reason for this tolerance to be so low, and that it is hard coded. But i guess you can test if there are any other issues. In that case i would use the car model for testing, as it can move slower.

[xxEoD2242]

@phil333 @GimpelZhang Yup, that was what I was talking about.

From my experience with AirSim, even with every optimization, you aren't going to get that time difference to be within Fusion tolerance. If AirSim is updated with faster image capture internally, then that is possible. But from how AirSim works under the hood, getting UE to actually generate the images is just not fast enough for 3 milliseconds of latency.

[phil333]

:(

[catproof]

I understand there is something fundamentally wrong with how the images are captured. But I don't see how setting the clock speed very low in simulation doesn't give an acceptable delay between when left images are taken and right images are taken. doesn't AirSim also have a RGBD camera sensor? That is ultimately quite similar to what stereo cameras have to offer.

[xxEoD2242]

@NickPerezCarletonUniversity if that works, then great. However, when working with Fusion, we raise the tolerance of the image difference. For our purposes, lowering the clock speed isn't an option due to processing requirements and time calculations.

Depth is great but when doing VIO, you need both camera images. Doing VIO from depth is possible but VINS Fusion doesn't operate off of that type of image.

[catproof]

"For our purposes, lowering the clock speed isn't an option due to processing requirements and time calculations." what do you mean by this? All I am saying is you can reduce how fast the simulation environment runs. https://microsoft.github.io/AirSim/settings.html#clockspeed

[xxEoD2242]

Reducing the clock speed means that the environment takes longer to update. We do real time trajectory generation that requires feedback that is close to what we do in the real world. Great for developing VIO but not for implementing on real world systems.