AirSim Crash due to msgpackrpc

[srivatsankrishnan]

trafficstars

Hi, I am seeing AirSim crash due to an Exception in msgpackrpc. I am seeing this issue from commits made from Jan 3rd to the latest version available on github. Here is the stack track:

ERROR:tornado.application:Uncaught exception, closing connection.
Traceback (most recent call last):
  File "C:\Python3.5\lib\site-packages\tornado\iostream.py", line 562, in wrapper
    return callback(*args)
  File "C:\Python3.5\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\Python3.5\lib\site-packages\msgpackrpc\transport\tcp.py", line 23, in on_read
    for message in self._unpacker:
  File "C:\Python3.5\lib\site-packages\msgpack\fallback.py", line 626, in __next__
    ret = self._unpack(EX_CONSTRUCT)
  File "C:\Python3.5\lib\site-packages\msgpack\fallback.py", line 579, in _unpack
    ret.append(self._unpack(EX_CONSTRUCT))
  File "C:\Python3.5\lib\site-packages\msgpack\fallback.py", line 600, in _unpack
    ret[key] = self._unpack(EX_CONSTRUCT)
  File "C:\Python3.5\lib\site-packages\msgpack\fallback.py", line 608, in _unpack
    obj = obj.decode(self._encoding, self._unicode_errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 0: invalid start byte
ERROR:tornado.application:Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x000001FC8B4A6F28>)
Traceback (most recent call last):
  File "C:\Python3.5\lib\site-packages\tornado\ioloop.py", line 605, in _run_callback
    ret = callback()
  File "C:\Python3.5\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\Python3.5\lib\site-packages\tornado\iostream.py", line 562, in wrapper
    return callback(*args)
  File "C:\Python3.5\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\Python3.5\lib\site-packages\msgpackrpc\transport\tcp.py", line 23, in on_read
    for message in self._unpacker:
  File "C:\Python3.5\lib\site-packages\msgpack\fallback.py", line 626, in __next__
    ret = self._unpack(EX_CONSTRUCT)
  File "C:\Python3.5\lib\site-packages\msgpack\fallback.py", line 579, in _unpack
    ret.append(self._unpack(EX_CONSTRUCT))
  File "C:\Python3.5\lib\site-packages\msgpack\fallback.py", line 600, in _unpack
    ret[key] = self._unpack(EX_CONSTRUCT)
  File "C:\Python3.5\lib\site-packages\msgpack\fallback.py", line 608, in _unpack
    obj = obj.decode(self._encoding, self._unicode_errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 0: invalid start byte
ERROR:tornado.application:Uncaught exception, closing connection.
Traceback (most recent call last):
  File "C:\Python3.5\lib\site-packages\tornado\iostream.py", line 562, in wrapper
    return callback(*args)
  File "C:\Python3.5\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\Python3.5\lib\site-packages\msgpackrpc\transport\tcp.py", line 24, in on_read
    self.on_message(message)
  File "C:\Python3.5\lib\site-packages\msgpackrpc\transport\tcp.py", line 29, in on_message
    raise RPCError("Invalid MessagePack-RPC protocol: message = {0}".format(message))
msgpackrpc.error.RPCError: Invalid MessagePack-RPC protocol: message = object_id
ERROR:tornado.application:Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x000001FCCB197048>)
Traceback (most recent call last):
  File "C:\Python3.5\lib\site-packages\tornado\ioloop.py", line 605, in _run_callback
    ret = callback()
  File "C:\Python3.5\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\Python3.5\lib\site-packages\tornado\iostream.py", line 562, in wrapper
    return callback(*args)
  File "C:\Python3.5\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\Python3.5\lib\site-packages\msgpackrpc\transport\tcp.py", line 24, in on_read
    self.on_message(message)
  File "C:\Python3.5\lib\site-packages\msgpackrpc\transport\tcp.py", line 29, in on_message
    raise RPCError("Invalid MessagePack-RPC protocol: message = {0}".format(message))
msgpackrpc.error.RPCError: Invalid MessagePack-RPC protocol: message = object_id

My msgpackrpc version is 0.4.1 AirSim version: current master (0d99b0c) OS: Windows 10 Python version: 3.5.4/3.6

I have successfully run previous versions of AirSim in a continuous loop for more than 4 days and haven't encountered this issue before.

Any insight or help is appreciated.

[srivatsankrishnan]

Hi @sytelus, Have you seen this error before? We aren't able to run AirSim continuously for more than a day and it limits our ability to use AirSim for training RL algorithms. The older version of AirSim (April 2018) is stable and we don't run into this issue. But we see that a lot of refactoring in the code and new features are added. It will be beneficial to have a stable version of latest AirSim.

[DevMMI]

I encountered a similar issue after running for Airsim for about 8 hours straight, my version is in line with commit SHA b8636806e70baee648810d7eab98325e6a4e8afe.

Error encountered:

ERROR:tornado.application:Uncaught exception, closing connection. Traceback (most recent call last): File "msgpack/_unpacker.pyx", line 479, in msgpack._unpacker.Unpacker._unpack UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 8: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/user/.local/lib/python3.5/site-packages/tornado/iostream.py", line 562, in wrapper return callback(*args) File "/home/user/.local/lib/python3.5/site-packages/tornado/stack_context.py", line 277, in null_wrapper return fn(*args, **kwargs) File "/home/user/.local/lib/python3.5/site-packages/msgpackrpc/transport/tcp.py", line 23, in on_read for message in self._unpacker: File "msgpack/_unpacker.pyx", line 554, in msgpack._unpacker.Unpacker.next File "msgpack/_unpacker.pyx", line 499, in msgpack._unpacker.Unpacker._unpack msgpack.exceptions.UnpackValueError: 'utf-8' codec can't decode byte 0x80 in position 8: invalid start byte ERROR:tornado.application:Exception in callback functools.partial(<function wrap..null_wrapper at 0x7f5a943c3378>) Traceback (most recent call last): File "msgpack/_unpacker.pyx", line 479, in msgpack._unpacker.Unpacker._unpack UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 8: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/user/.local/lib/python3.5/site-packages/tornado/ioloop.py", line 605, in _run_callback ret = callback() File "/home/user/.local/lib/python3.5/site-packages/tornado/stack_context.py", line 277, in null_wrapper return fn(*args, **kwargs) File "/home/user/.local/lib/python3.5/site-packages/tornado/iostream.py", line 562, in wrapper return callback(*args) File "/home/user/.local/lib/python3.5/site-packages/tornado/stack_context.py", line 277, in null_wrapper return fn(*args, **kwargs) File "/home/user/.local/lib/python3.5/site-packages/msgpackrpc/transport/tcp.py", line 23, in on_read for message in self._unpacker: File "msgpack/_unpacker.pyx", line 554, in msgpack._unpacker.Unpacker.next File "msgpack/_unpacker.pyx", line 499, in msgpack._unpacker.Unpacker._unpack msgpack.exceptions.UnpackValueError: 'utf-8' codec can't decode byte 0x80 in position 8: invalid start byte Exception in thread Thread-3: Traceback (most recent call last): File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner self.run() File "/usr/lib/python3.5/threading.py", line 862, in run self._target(*self._args, **self.kwargs) File "/home/user/AirSim/Airsim_RL/D4PG/Agent.py", line 147, in run s, r, done, , error, end_episode = self.env.act(a) File "/home/user/AirSim/Airsim_RL/utils/Environment.py", line 77, in act s, r_tmp, done, info, e, end_episode = self.env.step(action) File "/home/user/AirSim/Airsim_RL/simple_airgym/simple_airgym/envs/forest_env.py", line 37, in step collided, position, velocity = airgym.takeAction(action) File "/home/user/AirSim/Airsim_RL/simple_airgym/simple_airgym/envs/simple_sim_client.py", line 38, in takeAction collision_info = self.simGetCollisionInfo() File "/home/user/.local/lib/python3.5/site-packages/airsim/client.py", line 97, in simGetCollisionInfo return CollisionInfo.from_msgpack(self.client.call('simGetCollisionInfo', vehicle_name)) File "/home/user/.local/lib/python3.5/site-packages/msgpackrpc/session.py", line 41, in call return self.send_request(method, args).get() File "/home/user/.local/lib/python3.5/site-packages/msgpackrpc/future.py", line 43, in get raise self._error msgpackrpc.error.TimeoutError: Request timed out

[srivatsankrishnan]

@DevMohamedMI We didn't get a fix from AirSim but we have an infrastructure built around AirSim such that whenever AirSim fails, we detect it and restart all the components that crashed because of this. Maybe you can try that since that should allow your training to run for a longer duration.

[DevMMI]

Oh very interesting @srivatsankrishnan , are you listening for the PID externally or have you put some fault handling inside Airsim? And how did you re-setup your Airsim workspace and re-run it programmatically? Thanks for the reply.

[yzm1205]

Hey guys, I think I am facing a similar issue. Could you guys please guide me how to solve this issue?

Traceback (most recent call last): File "pg2.py", line 5, in import airsim File "C:\Users\lenovo\Anaconda3\lib\site-packages\airsim_init_.py", line 1, in from .client import * File "C:\Users\lenovo\Anaconda3\lib\site-packages\airsim\client.py", line 3, in from .utils import * File "C:\Users\lenovo\Anaconda3\lib\site-packages\airsim\utils.py", line 10, in from .types import * File "C:\Users\lenovo\Anaconda3\lib\site-packages\airsim\types.py", line 2, in import msgpackrpc #install as admin: pip install msgpack-rpc-python File "C:\Users\lenovo\Anaconda3\lib\site-packages\msgpackrpc_init_.py", line 6, in from msgpackrpc.server import Server File "C:\Users\lenovo\Anaconda3\lib\site-packages\msgpackrpc\server.py", line 81, in class _Responder: File "C:\Users\lenovo\Anaconda3\lib\site-packages\msgpackrpc\server.py", line 87, in _Responder def set_result(self, value, error=None, packer=msgpack.Packer()): AttributeError: module 'msgpack' has no attribute 'Packer'

Thanks

[XueZhihan]

Meet the same bug. Does anyone solve it during these 2 years?

[ShuboLiu]

Plus 1. I got exactly the same problem, Is there any way to solve it?

[mhtsbt]

I also encountered the same issue. It started occurring after I started requesting collision info from the Python API. This seems to be also the case mentioned by DevMMI

[WreckItTim]

bump.

It happens, seemingly randomly, after running for a long time. I had the same repo running AirSim across 4 different Linux servers. Occasionally one will crash, from any server, with this same exact error.