AirSim crashes when using PX4

[talha-opteran]

Bug report

• AirSim Version/#commit: v1.7.0
• UE/Unity version: Unreal 4.27.2
• autopilot version: PX4 v.1.13 or v.1.11
• OS Version: Ubuntu 20.04

What's the issue you encountered?

AirSim crashes intermittently when trying to launch using PX4 either through binaries or source. It is most evident when launching/exiting in quick succession. It will work 1/3 times. Error message below.

Settings

Default PX4 settings:

{
    "SettingsVersion": 1.2,
    "SimMode": "Multirotor",
    "ClockType": "SteppableClock",
    "Vehicles": {
        "PX4": {
            "VehicleType": "PX4Multirotor",
            "UseSerial": false,
            "LockStep": true,
            "UseTcp": true,
            "TcpPort": 4560,
            "ControlPortLocal": 14540,
            "ControlPortRemote": 14580,
            "Sensors":{
                "Barometer":{
                    "SensorType": 1,
                    "Enabled": true,
                    "PressureFactorSigma": 0.0001825
                }
            },
            "Parameters": {
                "NAV_RCL_ACT": 0,
                "NAV_DLL_ACT": 0,
                "COM_OBL_ACT": 1,
                "LPE_LAT": 47.641468,
                "LPE_LON": -122.140165
            }
        }
    }
}

How can the issue be reproduced?

1. Launch PX4: make px4_sitl_default none_iris
2. Launch AirSim using above settings
3. If it works correctly, exit and repeat steps1& 2. It will most certainly fail for the next ~3 attempts before working again.

Include full error message in text form

[2022.07.01-20.09.53:861][  0]LogTemp: Opening mavlink connection
Opening mavlink connection
[2022.07.01-20.09.53:861][  0]LogTemp: Disconnecting mavlink vehicle
Disconnecting mavlink vehicle
Waiting for mavlink vehicle...
Disconnecting mavlink vehicle
Waiting for TCP connection on port 4560, local IP 127.0.0.1
terminating with uncaught exception of type std::runtime_error: TcpClientPort socket bind failed with error: 98

Signal 6 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554 
CommonUnixCrashHandler: Signal=6
[2022.07.01-20.09.53:898][  0]LogCore: === Critical error: ===
Unhandled Exception: SIGABRT: abort() called

[2022.07.01-20.09.53:898][  0]LogCore: Fatal error!

0x00007f4ac6a7900b libc.so.6!gsignal(+0xcb)
0x00007f4ac6a58859 libc.so.6!abort(+0x12a)
0x000000000854d6fb ZhangJiajie!abort_message()
0x000000000854d60e ZhangJiajie!demangling_terminate_handler()()
0x000000000854d483 ZhangJiajie!std::__terminate(void (*)())()
0x000000000854cd06 ZhangJiajie!__cxxabiv1::failed_throw(__cxxabiv1::__cxa_exception*)()
0x000000000854cc9f ZhangJiajie!__cxa_throw(+0x6e)
0x0000000007700dc0 ZhangJiajie!TcpClientPort::TcpSocketImpl::accept(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [/datadrive2/AirSim/MavLinkCom/src/serial_com/TcpClientPort.cpp:153]
0x00000000076ea838 ZhangJiajie!mavlinkcom_impl::MavLinkConnectionImpl::acceptTcp(std::__1::shared_ptr<mavlinkcom::MavLinkConnection>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [/datadrive2/AirSim/MavLinkCom/src/impl/MavLinkConnectionImpl.cpp:96]
0x00000000076ad242 ZhangJiajie!mavlinkcom::MavLinkConnection::acceptTcp(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [/datadrive2/AirSim/MavLinkCom/src/MavLinkConnection.cpp:37]
0x0000000002d110f6 ZhangJiajie!msr::airlib::MavLinkMultirotorApi::createMavEthernetConnection(msr::airlib::AirSimSettings::MavLinkConnectionInfo const&) [/datadrive2/Marketplace/ZhangJiajie/Plugins/AirSim/Source/AirLib/include/vehicles/multirotor/firmwares/mavlink/MavLinkMultirotorApi.hpp:1297]
0x0000000002d0f4bc ZhangJiajie!msr::airlib::MavLinkMultirotorApi::connect_thread() [/datadrive2/Marketplace/ZhangJiajie/Plugins/AirSim/Source/AirLib/include/vehicles/multirotor/firmwares/mavlink/MavLinkMultirotorApi.hpp:964]
0x0000000002d157f8 ZhangJiajie!void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (msr::airlib::MavLinkMultirotorApi::*)(), msr::airlib::MavLinkMultirotorApi*> >(void*) [/datadrive/ue_425/ue-4.25.1-linux-debugeditor/Engine/Source/ThirdParty/Linux/LibCxx/include/c++/v1/thread:287]
0x00007f4ac74fc609 libpthread.so.0!UnknownFunction(0x8608)
0x00007f4ac6b55133 libc.so.6!clone(+0x42)

What's better than filing an issue? Filing a pull request :).

[jonyMarino]

Hi @talha-opteran ! Last time we needed to sync with PX4 to make it work, it was with version v1.12.0-beta1. Can you try with that on?

[talha-opteran]

No difference. It seems like there is a 10-15sec cooldown between connections

[jonyMarino]

Which are your hardware specs?

[talha-opteran]

Dell G15 CPU: i7-10870H GPU: RTX 3050Ti 4GB RAM: 32GB

[buvnswrn]

I guess the issue is not because of PX4-Airsim connection but Airsim settings.json itself. I wanted to use a different SITL controller called simple_airsim and this also had the same issue but in Unity i could see that the name of the Quadcopter used is SimpleFlight so i gave the same in the settings file under the VehicleType and i am able to use it as required, no crashes

[fengtudou]

Could you please confirm if this issue has been resolved?

[talha-opteran]

Not sure/ can't remember as I stopped using it actively in Linux. If you or anyone else is still encountering it then it likely still exists. I do recall using netstat to monitor the ports which remained in a disconnected state for a while after exiting the binary. Once the port(s) were no longer visible to netstat, then it would work again. This took ~30-60s