tractor icon indicating copy to clipboard operation
tractor copied to clipboard

RAE fails to lookup relayed remote non-builtin `Exception` during msg-delivery logging..

Open goodboy opened this issue 4 months ago • 0 comments

I've seen and been told about this one a few times now coming from a dependent project. It seems we've got a log-msg trying to render the .__repr__() on a relayed remote error and the local namespace lookup for that exc type by name fails in ._exceptions via the call chain,

RemoteActorError.pformat() -> RAE._mk_fields_str() -> RAE.is_inception() -> RAE.src_type() -> .get_err_type() :boom:

In this case the remote 3rd-party-lib's exc type which fails lookup is a trio_websocket._impl.DisconnectionTimeout.

This failed tractor._exceptions.get_err_type() call seems to actually result in an (unanticipated) crash, so it's pretty problematic XD

Here is a shortened tb from that case crashing a long running pikerd subactor..

Traceback (most recent call last):
    ...<N redacted frames>...

  File "/home/goodboy/repos/piker/piker/service/_actor_runtime.py", line 194, in open_pikerd
    open_piker_runtime(
    ~~~~~~~~~~~~~~~~~~^

    ...<N redacted lines>...

  File "/home/goodboy/repos/piker/piker/service/_actor_runtime.py", line 108, in open_piker_runtime
    tractor.open_root_actor(
    ~~~~~~~~~~~~~~~~~~~~~~~^


    ...<17 lines>...
        **tractor_kwargs,
        ^^^^^^^^^^^^^^^^^
    ) as actor,
    ^
  File "/usr/lib/python3.13/contextlib.py", line 235, in __aexit__
    await self.gen.athrow(value)
  File "/home/goodboy/repos/tractor/tractor/_root.py", line 481, in open_root_actor
    collapse_eg(),
    ~~~~~~~~~~~^^
  File "/usr/lib/python3.13/contextlib.py", line 235, in __aexit__
    await self.gen.athrow(value)
  File "/home/goodboy/repos/tractor/tractor/trionics/_beg.py", line 161, in collapse_eg
    raise exc from None
  File "/home/goodboy/repos/tractor/tractor/_runtime.py", line 1733, in async_main
    raise internal_err
  File "/home/goodboy/repos/tractor/tractor/_runtime.py", line 1548, in async_main
    collapse_eg(),
    ~~~~~~~~~~~^^
  File "/usr/lib/python3.13/contextlib.py", line 235, in __aexit__
    await self.gen.athrow(value)
  File "/home/goodboy/repos/tractor/tractor/trionics/_beg.py", line 161, in collapse_eg
    raise exc from None
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_highlevel_serve_listeners.py", line 34, in _run_handler
    await handler(stream)
  File "/home/goodboy/repos/tractor/tractor/ipc/_server.py", line 447, in handle_stream_from_peer
    ) = await _rpc.process_messages(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        chan=chan,
        ^^^^^^^^^^
    )
    ^
  File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 1056, in process_messages
    await actor._deliver_ctx_payload(
    ...<3 lines>...
    )
  File "/home/goodboy/repos/tractor/tractor/_runtime.py", line 646, in _deliver_ctx_payload
    return await ctx._deliver_msg(msg)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/tractor/tractor/_context.py", line 1844, in _deliver_msg
    f'{pformat(re)}\n'
       ~~~~~~~^^^^
  File "/usr/lib/python3.13/pprint.py", line 62, in pformat
    underscore_numbers=underscore_numbers).pformat(object)
                                           ~~~~~~~^^^^^^^^
  File "/usr/lib/python3.13/pprint.py", line 161, in pformat
    self._format(object, sio, 0, 0, {}, 0)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/pprint.py", line 178, in _format
    rep = self._repr(object, context, level)
  File "/usr/lib/python3.13/pprint.py", line 458, in _repr
    repr, readable, recursive = self.format(object, context.copy(),
                                ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
                                            self._depth, level)
                                            ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/pprint.py", line 471, in format
    return self._safe_repr(object, context, maxlevels, level)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/pprint.py", line 632, in _safe_repr
    rep = repr(object)
  File "/home/goodboy/repos/tractor/tractor/_exceptions.py", line 627, in pformat
    fields: str = self._mk_fields_str(
                  ~~~~~~~~~~~~~~~~~~~^
        _body_fields
        ^^^^^^^^^^^^
        +
        ^
        self.extra_body_fields,
        ^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/goodboy/repos/tractor/tractor/_exceptions.py", line 524, in _mk_fields_str
    and not self.is_inception()
            ~~~~~~~~~~~~~~~~~^^
  File "/home/goodboy/repos/tractor/tractor/_exceptions.py", line 591, in is_inception
    self.src_type
  File "/home/goodboy/repos/tractor/tractor/_exceptions.py", line 427, in src_type
    raise TypeError(
    ...<3 lines>...
    )
TypeError: Failed to lookup src error type with `tractor._exceptions.get_err_type()` :
DisconnectionTimeout
( ^^^ this exc was collapsed from a group ^^^ )

How to fix?

  • [ ] should we just drop the TypeError raise from RemoteActorError.src_type whenever the RAE is reading/calling .src_type from a .is_inception() call??
    • could impl via method flag?

goodboy avatar Oct 07 '25 14:10 goodboy