Remove SimMode and support different vehicle types

[alonfaraj]

Fixes: https://github.com/microsoft/AirSim/issues/506 Fixes: https://github.com/microsoft/AirSim/issues/1382 Fixes: https://github.com/microsoft/AirSim/issues/1777 Fixes: https://github.com/microsoft/AirSim/issues/2990 Fixes: https://github.com/microsoft/AirSim/issues/4084 Fixes: https://github.com/microsoft/AirSim/issues/4110

About

As a follow up to the great work of @rajat2004 (https://github.com/microsoft/AirSim/issues/4110), this PR is intended support different vehicle types in AirSim and remove SimMode from settings.json file.

There are probably few approaches to achieve this goal.
I chose the one which IMO should be the easiest to implement and maintain.

We have already started working with this version for our custom needs so I thought it would be worth sharing with the community.
This PR should be tested much more, but it already work with different vehicle types - in Unreal and ROS2 (ROS1 modified as well but haven't tested yet).
Any feedbacks/thoughts/issues are welcome.

Changes:

• SimMode is no longer required and ignored
• SensorsDefault changed to SensorsDefaultCar and SensorsDefaultMultirotor
• Record file was split into 3 files - airsim_rec_cv.txt, airsim_rec_car.txt, airsim_rec_multirotor.txt and saved into the same folder
• ApiServerPort and its default value changed to ApiServerPortCV (41451), ApiServerPortCar(41461), ApiServerPortMultirotor(41471)

How Has This Been Tested?

Tested with Unreal, python API and ROS2, with various combinations and features. Attached an example script and settings to initialize with 3 different types, send commands and spawn another 3 vehicles.

example settings.json

{
	"SettingsVersion": 1.2,

	"Vehicles": {
		"test1_car": {
			"VehicleType": "PhysXCar",
			"X": 0, "Y": 7, "Z": 0		
		},
		"test2_cv": {
			"VehicleType": "ComputerVision",
			"X": 0, "Y": 3, "Z": 0		
		},
		"test3_drone": {
			"VehicleType": "SimpleFlight"
		}
	}

  }

example script

import setup_path 
import airsim
import cv2

cv = airsim.VehicleClient()
cv.confirmConnection()

drone = airsim.MultirotorClient()
drone.confirmConnection()
drone.enableApiControl(True)

car = airsim.CarClient()
car.confirmConnection()
car.enableApiControl(True)


drone.takeoffAsync()

car_controls = airsim.CarControls()
car_controls.steering = 0.0
car_controls.throttle = 1.0
car_controls.brake = 0.0
car.setCarControls(car_controls)

pose = airsim.Pose(airsim.Vector3r(0, -2, 0),
                       airsim.Quaternionr(0, 0, 0, 0))
drone.simAddVehicle("drone2", "SimpleFlight", pose)

pose = airsim.Pose(airsim.Vector3r(2, 0, 0),
                       airsim.Quaternionr(0, 0, 0, 0))
car.simAddVehicle("car2", "PhysXCar", pose)

pose = airsim.Pose(airsim.Vector3r(0, 5, 0),
                       airsim.Quaternionr(0, 0, 0, 0))
cv.simAddVehicle("cv2", "ComputerVision", pose)


camera_name = "0"
image_type = airsim.ImageType.Scene
while True:
    rawImage = cv.simGetImage(camera_name, image_type)
    if not rawImage:
        continue
    png = cv2.imdecode(airsim.string_to_uint8_array(rawImage), cv2.IMREAD_UNCHANGED)
    
    cv2.imshow("AirSim", png)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
cv2.destroyAllWindows() 

Screenshots (if appropriate):

https://user-images.githubusercontent.com/15960497/152807667-147a6b54-d219-42cf-b207-f6e92fada41a.mp4

recording files: airsim_rec_cv.txt airsim_rec_multirotor.txt airsim_rec_car.txt

[alonfaraj]

Thanks @rajat2004! Hope you well. I will address your review soon.

[alonfaraj]

@rajat2004 Thank again for the review. About removing the external cameras - I think it's a good idea. It makes the API simpler and remove an extra code. Let's make sure everything works as expected in this PR before we make this change.

[alonfaraj]

Did a bit more thorough review. One thing was that it's probably better to remove the kSimModeType* constants from AirSimSettings. Also, with the current change, what happens when the settings file is empty, does the prompt and creation of default car or multirotor work?

You right, I left it for now because simmode_getter() return a kSimModeType*, it probably should be modified as well. If there is no settings file - the prompt works like before.

Update - Fixed, kSimModeType* removed

[rajat2004]

Just tried compiling and running this for the first time locally, crashes during startup though using the settings in the description (and also with just a single multirotor) -

Stacktrace

Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554 
CommonUnixCrashHandler: Signal=11
Malloc Size=43626 LargeMemoryPoolOffset=109210 
Malloc Size=63410 LargeMemoryPoolOffset=172642 
Malloc Size=43626 LargeMemoryPoolOffset=216298 
Malloc Size=63410 LargeMemoryPoolOffset=279730 
Malloc Size=43626 LargeMemoryPoolOffset=323386 
Malloc Size=63462 LargeMemoryPoolOffset=386870 
[2022.02.27-13.59.30:441][ 91]LogCore: === Critical error: ===
Unhandled Exception: SIGSEGV: invalid attempt to write memory at address 0x0000000000000080

[2022.02.27-13.59.30:441][ 91]LogCore: Fatal error!

0x00007f6d8f60082d libUE4Editor-AirSim.so!ASimModeBase::BeginPlay() [/home/rajat/AirSim/Unreal/Environments/Blocks/Plugins/AirSim/Source/SimMode/SimModeBase.cpp:136]
0x00007f6d8f617629 libUE4Editor-AirSim.so!ASimModeWorldMultiRotor::BeginPlay() [/home/rajat/AirSim/Unreal/Environments/Blocks/Plugins/AirSim/Source/Vehicles/Multirotor/SimModeWorldMultiRotor.cpp:21]
0x00007f6e54ad50dc libUE4Editor-Engine.so!AActor::DispatchBeginPlay(bool) [/home/rajat/UnrealEngine/UE_4.25/Engine/Source/Runtime/Engine/Private/Actor.cpp:3395]
0x00007f6e54ad3d4c libUE4Editor-Engine.so!AActor::PostActorConstruction() [/home/rajat/UnrealEngine/UE_4.25/Engine/Source/Runtime/Engine/Private/Actor.cpp:3238]
0x00007f6e54ad18aa libUE4Editor-Engine.so!AActor::FinishSpawning(FTransform const&, bool, FComponentInstanceDataCache const*) [/home/rajat/UnrealEngine/UE_4.25/Engine/Source/Runtime/Engine/Private/Actor.cpp:3137]
0x00007f6e54acfc03 libUE4Editor-Engine.so!AActor::PostSpawnInitialize(FTransform const&, AActor*, APawn*, bool, bool, bool) [/home/rajat/UnrealEngine/UE_4.25/Engine/Source/Runtime/Engine/Private/Actor.cpp:3079]
0x00007f6e5544be44 libUE4Editor-Engine.so!UWorld::SpawnActor(UClass*, FTransform const*, FActorSpawnParameters const&) [/home/rajat/UnrealEngine/UE_4.25/Engine/Source/Runtime/Engine/Private/LevelActor.cpp:509]
0x00007f6e5544cbdd libUE4Editor-Engine.so!UWorld::SpawnActor(UClass*, FVector const*, FRotator const*, FActorSpawnParameters const&) [/home/rajat/UnrealEngine/UE_4.25/Engine/Source/Runtime/Engine/Private/LevelActor.cpp:303]
0x00007f6d8f5fafad libUE4Editor-AirSim.so!ASimHUD::createAllSimModes() [/home/rajat/AirSim/Unreal/Environments/Blocks/Plugins/AirSim/Source/SimHUD/SimHUD.cpp:284]
0x00007f6d8f5fa3ce libUE4Editor-AirSim.so!ASimHUD::BeginPlay() [/home/rajat/AirSim/Unreal/Environments/Blocks/Plugins/AirSim/Source/SimHUD/SimHUD.cpp:29]

I think recording needs more thought, earlier, we had a single (static) RecordingThread handling all the vehicles (which were of the same type such as Multirotor due to simmode restrictions). This works since we don't want multiple writers for the same file.

In the current state of the PR, we're creating a new RecordingThread for each vehicle which is wrong (I guess it worked in the initial testing since there was a single type of each vehicle). I think we would need a single instance of RecordingThread per-vehicle type, i.e. 1 for each Multirotor, Car, CV. And it would handle all the vehicles of that type.

One way I can think of is to move the recording_thread_ to each of the subclass SimModeMultirotor, etc, and make that static, The different recording methods will need to be overridden in the SimMode* subclasses, and for e.g. for startRecording it could pass the relevant vehicles to RecordingThread. What do you think?

[alonfaraj]

@rajat2004 Sorry about that, seems like something messed up during the recent commits. It's not just the recording changes. Can you please go back and try checkout 7be0a72?

[zimmy87]

The Windows build check is failing on this PR due to an outdated .yml file which has been fixed by #4377. To pick up this fix, please rebase to the latest master.

[rajat2004]

Also, had some time so did a Windows packaging to see why the CI is failing, getting the following error -

UATHelper: Packaging (Windows (64-bit)):   LogInit: Display: Warning/Error Summary (Unique only)
UATHelper: Packaging (Windows (64-bit)):   LogInit: Display: -----------------------------------
UATHelper: Packaging (Windows (64-bit)):   LogInit: Display: LogBlueprint: Error: [Compiler BP_LevelLoadButton] In use pin  Return Value  no longer exists on node  Get Sim Mode . Please refresh node or break links to remove pin. from Source: /AirSim/HUDAssets/BP_LevelLoadButton.BP_LevelLoadButton
UATHelper: Packaging (Windows (64-bit)):   LogInit: Display: LogBlueprint: Error: [Compiler BP_LevelLoadButton] Could not find a function named "getSimMode" in 'SimModeBase'.
UATHelper: Packaging (Windows (64-bit)):   Make sure 'SimModeBase' has been compiled for  Get Sim Mode from Source: /AirSim/HUDAssets/BP_LevelLoadButton.BP_LevelLoadButton

[alonfaraj]

Also, had some time so did a Windows packaging to see why the CI is failing, getting the following error -

UATHelper: Packaging (Windows (64-bit)):   LogInit: Display: Warning/Error Summary (Unique only)
UATHelper: Packaging (Windows (64-bit)):   LogInit: Display: -----------------------------------
UATHelper: Packaging (Windows (64-bit)):   LogInit: Display: LogBlueprint: Error: [Compiler BP_LevelLoadButton] In use pin  Return Value  no longer exists on node  Get Sim Mode . Please refresh node or break links to remove pin. from Source: /AirSim/HUDAssets/BP_LevelLoadButton.BP_LevelLoadButton
UATHelper: Packaging (Windows (64-bit)):   LogInit: Display: LogBlueprint: Error: [Compiler BP_LevelLoadButton] Could not find a function named "getSimMode" in 'SimModeBase'.
UATHelper: Packaging (Windows (64-bit)):   Make sure 'SimModeBase' has been compiled for  Get Sim Mode from Source: /AirSim/HUDAssets/BP_LevelLoadButton.BP_LevelLoadButton

Thanks! I didn't notice that. I fixed it by get all SimModeBase actors in the BP and iterate them where needed.

UPDATE - Check still fail, not sure why. I was able to package blocks on windows without any errors.

[alonfaraj]

Again, sorry for the late reply. Did some small basic runs on Ubuntu 18.04, UE 4.25, working nicely!

In the current state of the PR, we're creating a new RecordingThread for each vehicle which is wrong (I guess it worked in the initial testing since there was a single type of each vehicle). I think we would need a single instance of RecordingThread per-vehicle type, i.e. 1 for each Multirotor, Car, CV. And it would handle all the vehicles of that type.

Thinking on this further, my above comment is probably incorrect. What we actually have is 3 servers, once for each "SimMode". So we're not creating a RecordingThread per vehicle but only 3 which is fine. Please ignore the rest of the comment

One place this might still cause an issue is where we have startRecording() API used from 2 different clients (like Car, Multirotor). In RecordingThread::startRecording(), there's a check assert(!isRecording()) which would crash the sim in this case. But with the new changes, for different clients, it wouldn't work since the instances will be different, basically leading to double recording. Need to think about whether this can be handled, or should it be handled at all.

I'm not sure about it. I made a quick test using the API. Create 3 clients, start record with each client. Everything works as excepted. What do you mean by double recording?

[yvan233]

Thanks for your work. This is the functionality I need now. I compiled and ran the latest branch locally and reproduced example script.py without any bugs. But when I test the drone's moveToPositionAsync function like below. The drone will be abnormal and cannot fly to the designated position.

import setup_path 
import airsim
import os

client = airsim.MultirotorClient()
client.confirmConnection()
client.enableApiControl(True)
client.armDisarm(True)

client.takeoffAsync()  
client.moveToPositionAsync(0, 0, -4, 1).join()
client.moveToPositionAsync(5, 5, -4, 1).join()

Do you know what is causing this abnormality? Thanks!

[yvan233]

Thanks for your work. This is the functionality I need now. I compiled and ran the latest branch locally and reproduced example script.py without any bugs. But when I test the drone's moveToPositionAsync function like below. The drone will be abnormal and cannot fly to the designated position.

import setup_path 
import airsim
import os

client = airsim.MultirotorClient()
client.confirmConnection()
client.enableApiControl(True)
client.armDisarm(True)

client.takeoffAsync()  
client.moveToPositionAsync(0, 0, -4, 1).join()
client.moveToPositionAsync(5, 5, -4, 1).join()

Do you know what is causing this abnormality? Thanks!

I have found the problem. The clock type needs to be set in settings.json. like this:

"ClockType": "ScalableClock"

[dzywater]

Is it possible to let each client have respective api control status. Currently, one client disable the api control, all clients cannot control. So hope to let server give each client a control status record.

[A-Dings]

I cloned your remove-simmode branch , ran the example script with the example setting.json you provided on Ubuntu 18.04, UE 4.25. I got this error: Traceback (most recent call last): File "hello.py", line 10, in drone.enableApiControl(True) msgpackrpc.error.RPCError: rpclib: function 'enableApiControl' (called with 2 arg(s)) threw an exception. The exception contained this information: Vehicle API for '' is not available. This could either because this is simulation-only API or this vehicle does not exist. Could you please help look at this issue?

[sakurayiannie]

I cloned your remove-simmode branch , ran the example script with the example setting.json you provided on Ubuntu 18.04, UE 4.25. I got this error: Traceback (most recent call last): File "hello.py", line 10, in drone.enableApiControl(True) msgpackrpc.error.RPCError: rpclib: function 'enableApiControl' (called with 2 arg(s)) threw an exception. The exception contained this information: Vehicle API for '' is not available. This could either because this is simulation-only API or this vehicle does not exist. Could you please help look at this issue?

Hi！I also encountered this problem, is there any solution?

[alonfaraj]

Hi @sakurayiannie @A-Dings, I haven't tested it on ubuntu but I don't see any reason it will cause the error. Did you run the script clean_rebuild? There are modifications in AirLib, not only the unreal side

[sakurayiannie]

你好@sakurayiannie @A-Dings， 我没有在 ubuntu 上测试过，但我看不出有什么原因会导致错误。 你运行脚本了clean_rebuild吗？ AirLib有修改，不只是虚幻的一面

Thank you for your reply! I can run your remove-simmode project by c++ API (on Win10), while python client returned the same problem ('enableApiControl') as A-Dings encountered. I ran the script clean_rebuild.bat, it returned that "The system cannot find the specified file".

[hjp10010]

Can I only use his@alonfaraj modified old version of airsim to simulate different types of vehicles and UAVs at the same time? It seems that the new version of airsim does not have this function. In fact, I also encountered some problems with his modified old version of airsim.

[alonfaraj]

Hi everyone @hjp10010 @sakurayiannie, I haven't use this branch heavily back then so there are probably some issues. It's not fully tested and airsim is no longer maintained so I didn't work on this branch any further. It also use pretty old ue4 version, maybe 4.25 so I can't test it at the moment. It worked for me (and for others) with basics features, recording and api, as you can see in the video.

Sorry I can't help you more.

I think colosseum is the right repo to continue this feature development in case the maintainers interested. I would be glad to help.

[xxEoD2242]

Just leaving the link here for Colosseum:

Colosseum

[hjp10010]

Thank you for your reply. I would like to try the scheme you recommended

[sakurayiannie]

Hi everyone @hjp10010 @sakurayiannie, I haven't use this branch heavily back then so there are probably some issues. It's not fully tested and airsim is no longer maintained so I didn't work on this branch any further. It also use pretty old ue4 version, maybe 4.25 so I can't test it at the moment. It worked for me (and for others) with basics features, recording and api, as you can see in the video.

Sorry I can't help you more.

I think colosseum is the right repo to continue this feature development in case the maintainers interested. I would be glad to help.

Thank you for your contribution to AirSim, my project need to work on air-ground coordination robots, so your work really helped me a lot. I appreciate that. :)

[fellowchrononaut]

I am using the remove-simmode repository and I am facing the following issue: Whenever I launch the airsim_node.launch file, Unreal crashes with an access violation error in line 533 of SimModeBase.cpp where I think the getImageCapture () is returning a null pointer. I've added an image of the exception below.

And here is the call stack:

I am running the remove-simmode fork on Unreal 4.27.2 with the airsim_ros_pkgs (ROS 1) running on Ubuntu 20.04 on WSL2.

I have tested with the main Airsim repository, also with the Colosseum repository(with UE 5.1). All of them work fine with the ros package. I can get the airsim_node to work and I can get images. But this issue is seen only with the remove-simmode fork. The other topics from the airsim_node are received well if I isolate the images by setting the "update_airsim_img_response_every_n_sec" parameter in the airsim_node.launch file to a high value. So I assume it's an issue with just the image requests.

Can someone please help me debug this issue because I want this feature (Drone and Car in the same environment) for my project?

Thanks in advance

[dzywater]

Fixes: #506 Fixes: #1382 Fixes: #1777 Fixes: #2990 Fixes: #4084 Fixes: #4110

About

As a follow up to the great work of @rajat2004 (#4110), this PR is intended support different vehicle types in AirSim and remove SimMode from settings.json file.

There are probably few approaches to achieve this goal. I chose the one which IMO should be the easiest to implement and maintain.

We have already started working with this version for our custom needs so I thought it would be worth sharing with the community. This PR should be tested much more, but it already work with different vehicle types - in Unreal and ROS2 (ROS1 modified as well but haven't tested yet). Any feedbacks/thoughts/issues are welcome.

Changes:

• SimMode is no longer required and ignored
• SensorsDefault changed to SensorsDefaultCar and SensorsDefaultMultirotor
• Record file was split into 3 files - airsim_rec_cv.txt, airsim_rec_car.txt, airsim_rec_multirotor.txt and saved into the same folder
• ApiServerPort and its default value changed to ApiServerPortCV (41451), ApiServerPortCar(41461), ApiServerPortMultirotor(41471)

How Has This Been Tested?

Tested with Unreal, python API and ROS2, with various combinations and features. Attached an example script and settings to initialize with 3 different types, send commands and spawn another 3 vehicles.

example settings.json example script

Screenshots (if appropriate):

clip.mp4

recording files: airsim_rec_cv.txt airsim_rec_multirotor.txt airsim_rec_car.txt

Hi, @alonfaraj , airsim seems active again, we can continue this topic.

[alonfaraj]

@dzywater Do you mean colosseum? It's not so active as far as I can tell. This feature requires some more work and tests which I don't think we can get.

[xxEoD2242]

@dzywater Do you mean colosseum? It's not so active as far as I can tell. This feature requires some more work and tests which I don't think we can get.

@alonfaraj Is it possible to port some of this work to Colosseum? I have another approach I've been working on but it's a behemoth of a PR that I'm attempting to break down.

@dzywater AirSim has been abandoned by Microsoft. Colosseum is our effort to continue the open source project with UE5

[alonfaraj]

@xxEoD2242 sure. Feel free

[dzywater]

behemoth

Could you import this pr to colosseum? Afterwards, we maintain colosseum together.

[Willis814]

I encountered some problems. When I opened the block environment after running build.cmd, an error occurred: Severity Code Description Item File Line Suppressed Status Details Error Unable to instantiate module 'AirSim': System.IO.DirectoryNotFoundException: Part of the path 'D:\Unreal Projects\Test\Plugins\AirSim\Source\AirLib\deps\MotionCore' could not be found. Test D:\Unreal Projects\Test\Intermediate\ProjectFiles\UnrealBuildTool 1 What should I do to solve this problem? Thanks for the answer!

[JeroZe]

这是来自QQ邮箱的假期自动回复邮件。   您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。

[Willis814]

我正在使用remove-simmode存储库，并且面临以下问题： 每当我启动airsim_node.launch文件时，Unreal都会崩溃，并在SimModeBase.cpp的第533行中出现访问冲突错误，我认为getImageCapture（）返回null指针。我在下面添加了异常的图像。

这是调用堆栈：

我正在 Unreal 4.27.2 上运行remove-simmode fork，并在 WSL2 上的 Ubuntu 20.04 上运行airsim_ros_pkgs (ROS 1)。

我已经使用主 Airsim 存储库以及 Colosseum 存储库（使用 UE 5.1）进行了测试。它们都与 ros 包一起工作得很好。我可以让airsim_node 工作并且可以获取图像。但只有在使用remove-simmode 分支时才会出现此问题。如果我通过将airsim_node.launch 文件中的“update_airsim_img_response_every_n_sec”参数设置为较高值来隔离图像，则来自airsim_node 的其他主题会得到很好的接收。所以我认为这只是图像请求的问题。

有人可以帮我调试这个问题吗，因为我想要为我的项目提供此功能（无人机和汽车在同一环境中）？

提前致谢

Have you solved this problem? I want to know what causes the error?🥺