on_data_available provides a reader that is None

[matte1]

trafficstars

I'm attempting to do dynamic topic discovery through the BuiltinTopicReader. When I set my configuration file to be multicast and between two devices on a subnet everything works as expected. When I try to do the same thing using "lo" and no multicast instead the listener gets None instead of a data reader.

I'm not sure if I'm just going about this wrong completely or if this is an actual bug...

class TopicListener(Listener):
    """Receive topic information from the builtin reader."""

    def __init__(self) -> None:
        """Init."""
        Listener.__init__(self)

    def on_data_available(self, reader: BuiltinDataReader) -> None:
        """Read topic information and store it in the mapping."""
        if not reader:
          print("Failed to get reader")

I've played around with the tracing and haven't found anything that seemed helpful. I.e. info/warning/severe were completely empty.

[eboasson]

This doesn't make any obvious sense, except ... @thijsmie, could it be the case that the listener is installed when the reader is created and that the listener gets invoked before dds_create_reader has returned and that consequently the reader handle passed into the callback doesn't work/can't be translated into a python object yet?

[thijsmie]

Looks like that is exactly what happens @eboasson:

from cyclonedds.core import Listener
from cyclonedds.builtin import BuiltinDataReader, BuiltinTopicDcpsTopic
from cyclonedds.idl import IdlStruct
from cyclonedds.topic import Topic
from cyclonedds.sub import DataReader
from cyclonedds.domain import DomainParticipant


class Dummy(IdlStruct):
    nonce: int

class TopicListener(Listener):
    """Receive topic information from the builtin reader."""

    def __init__(self) -> None:
        """Init."""
        Listener.__init__(self)

    def on_data_available(self, reader) -> None:
        """Read topic information and store it in the mapping."""
        if not reader:
          print("- Failed to get reader")



print("Making DomainParticipant")
dp = DomainParticipant()
print(f'DomainParticipant is initialized')
print("Making Topic")
tp = Topic(dp, 'ya', Dummy)
print(f'Topic is initialized')
print("Making DataReader")
dr = DataReader(dp, tp)
print(f'DataReader is initialized')
print("Making BuiltinDataReader")
bdr = BuiltinDataReader(dp, BuiltinTopicDcpsTopic, listener=TopicListener())
print(f'BuiltinDataReader is initialized')

This prints:

Making DomainParticipant
DomainParticipant is initialized
Making Topic
Topic is initialized
Making DataReader
DataReader is initialized
Making BuiltinDataReader
- Failed to get reader
BuiltinDataReader is initialized

This can be semi-fixed by moving all the dds_create_* calls to raw C and not release the GIL like ctypes does. Then we still have a race condition between the return value of the creation call being inserted into the entity map and the listener, so if there is some IO in between or GC the GIL might be released and we end up in the same situation. Maybe we need entity_enable in the end so the listener is not invoked till the language backend is ready?

[matte1]

Thanks for taking a look!

[matte1]

Is there any easy fix for this that I could use to get unblocked?

[thijsmie]

I don't think there is an easy fix no. You can map the C function dds_domain_set_deafmute to to deafen the domain while you create the reader+listener and then undeafen it to avoid any listeners being called during startup which would work but I would not consider "easy". You can also modify the listener source so it always gives you the reader reference instead of the lookup, then you could schedule a callback later when the init is finish. Easiest is to not use listeners at all (they are not performant anyway with the python GIL situation) and use a waitset instead.

[matte1]

Waitsets worked nicely! Should I close this?

[eboasson]

Hi @matte1, I am happy to hear that the waitsets do the job for you.

As to your question: of course you should close this! Clean desk, clean inbox, zero bugs and zero issues and all that 🤣

But more realistically speaking, the title of the issue is fine description of the symptom directly associated with the root cause (listeners can be invoked before the Python object exists) and may well be useful to others while the underlying issues get addressed. (And as you may have noticed, @thijsmie and @willstott101 have already put in some hard work to improve the behaviour of the listeners.)