LightStreamer disconnects after about 2 hours

[RRMXkun]

This was already raised years ago in #84 and a possible fix was made in #113 but the issue still occurs. I am creating a new ticket so it can be investigated again, perhaps a different approach is needed to finally fix this issue. Pinging @sacharbit who provided the initial fix.

This is the error I am seeing:

ERROR:trading_ig.lightstreamer:Communication error

Will re-run with logging in DEBUG mode, wait... 2 hours and see if any more info is reported.

[RRMXkun]

I made some progress on my implementation of a heartbeat monitor in stream.py (IGStreamService). What it does is if the heartbeat stops pinging for more than 10 seconds, it will assume connection dropped, and retry to connect. To do so it will need to re-subscribe to all subscriptions once connection is re-established. Issue with this I think is the stream update thread is re-spawned and therefore so are listener methods. I can see two ways of implementing this in a non-disruptive way:

1. Have a method in IGStreamService e.g. connection_alive() that returns False in case of a drop in connection. This requires the main thread to have a while loop i.e. while stream.connection_alive(). It's good in that it enables the user to choose how to handle a disconnect and re-subscribe if they so wish. I added a resubscribe_all() method to stream.py to facilitate this
2. Automatically reconnect, but doing so from stream.py won't work for the reason mentioned above. I think we'd need to intercept a disconnection in lightsreamer.py's _receive() but I'm not too keen to touch that since it's obscure at best and the chances of unexpected behaviour are high

Any thoughts @oliverpolden @bug-or-feature @femtotrader ? Good thing is, since the (old) code samples encouraged to reference lighstreamer.py's subscribe() directly, adding the above will not break existing code. However going forward it would require to use stream.py's subscribe method instead since it keeps track of active subscriptions, enabling it to resubscribe_all if required.

[RRMXkun]

Also pinging @sacharbit who PRd the original fix

[sacharbit]

It's been 2 or 3 years since I used that library so I don't really remember but I know I've ran into the same issue during the weekend since there was no data for a long time. But it didn't bother me because in the weekend there's no data anyway and I was using it for very liquid products so there's always new data coming in less than 10 secs. I guess for more exotic ones, you'll need to have a workaround. Can't really help you with that, it took me waaaaay too long to figure out that fix that i've PRed and couldn't find anything else!

[bug-or-feature]

Sorry - I don't use the lightstreamer code

[BlayneMartin]

@RRMXkun - Did you ever come up with a working solution to this issue?

[RRMXkun]

@RRMXkun - Did you ever come up with a working solution to this issue?

yes, but it's not perfect in that it requires the main thread to loop. I basically have a connection_alive() method in IGStreamService which switches to False and throws an exception if the Lightstreamer Heartbeat stops responding for 5 seconds. That means the main thread needs to be running a while loop to check for the value returned by connection_alive(). However the plus side is, the user is able to catch that exception and decide what to do.

I didn't want to push the code because I haven't been able to test it for long enough. I wanted to add a nice auto reconnect option but it's tricky to test.

[BlayneMartin]

@RRMXkun - Did you ever come up with a working solution to this issue?

yes, but it's not perfect in that it requires the main thread to loop. I basically have a connection_alive() method in IGStreamService which switches to False and throws an exception if the Lightstreamer Heartbeat stops responding for 5 seconds. That means the main thread needs to be running a while loop to check for the value returned by connection_alive(). However the plus side is, the user is able to catch that exception and decide what to do.

I didn't want to push the code because I haven't been able to test it for long enough. I wanted to add a nice auto reconnect option but it's tricky to test.

Thanks @RRMXkun I've been trying something similar but my 'keep alive' check is in the high level calling code and checks for a flag written to a database (could just be a file) - My check loop replaces the INPUT section of IG Markets Stream API sample from Femto. I have then modifiied Lighstreamer.py to write the flag to the database/file, if there is any type of error. This allows the check to be seperate from the main thread. It's not an ideal solution though and I'm still testing.

[RRMXkun]

@RRMXkun - Did you ever come up with a working solution to this issue?

yes, but it's not perfect in that it requires the main thread to loop. I basically have a connection_alive() method in IGStreamService which switches to False and throws an exception if the Lightstreamer Heartbeat stops responding for 5 seconds. That means the main thread needs to be running a while loop to check for the value returned by connection_alive(). However the plus side is, the user is able to catch that exception and decide what to do. I didn't want to push the code because I haven't been able to test it for long enough. I wanted to add a nice auto reconnect option but it's tricky to test.

Thanks @RRMXkun I've been trying something similar but my 'keep alive' check is in the high level calling code and checks for a flag written to a database (could just be a file) - My check loop replaces the INPUT section of IG Markets Stream API sample from Femto. I have then modifiied Lighstreamer.py to write the flag to the database/file, if there is any type of error. This allows the check to be seperate from the main thread. It's not an ideal solution though and I'm still testing.

I see, has your additional logging provided any insights as to what might be happening? From my end I've got things working relatively, I think having an exception thrown on the main thread is actually the right thing to do since it enables handling as per the user's wish. Of course being able to silently reconnect would be better, but the only way I managed to do that so far is by re-subscribing everything. Problem is this spawns a new set of threads, so any reference made to the old, now-dead ones result in a Nonetype error. Once I manage to get that bit to work I will PR.

If you've made some progress your end please share, you may have come up with a better solution

[bug-or-feature]

I'm closing this, it's old. The latest release uses the official Lightstreamer Python client SDK, which does auto-reconnections, auto-resubscriptions. See #302