alpacahq
[QUESTION]: Is it possible to clear connections on server after a network outage?
Is there an existing issue for this?
- [X] I have searched the existing issues
What is your question?
Hi. I'm using WithReconnect to reconnect to the streaming endpoints after a network outage (simulated by disabling my network adapter). This does work, but I get the TooManyConnections status for about 90 seconds before the connection is authorized again (presumably because the alpaca servers recognize the dropped connection, and boots it out).
The question is - is it possible to reduce this waiting time somehow?
22/09/2022 14:25:09 Socket opened
22/09/2022 14:25:10 Connected: Authorized
- OnNext({"x":"","bp":19207.0,"ap":19209.0,"bs":1.0497,"as":4.6079,"c":[],"z":"","T":"q","S":"BTC/USD","t":"2022-09-22T12:25:12.8687621Z"})
- OnNext({"x":"","bp":19208.0,"ap":19209.0,"bs":0.475,"as":4.6079,"c":[],"z":"","T":"q","S":"BTC/USD","t":"2022-09-22T12:25:12.9484936Z"})
- OnNext({"x":"","bp":19208.0,"ap":19209.0,"bs":3.075,"as":4.6079,"c":[],"z":"","T":"q","S":"BTC/USD","t":"2022-09-22T12:25:13.0188649Z"})
...
- OnNext({"x":"","bp":1308.3,"ap":1308.6,"bs":57.653,"as":0.148,"c":[],"z":"","T":"q","S":"ETH/USD","t":"2022-09-22T12:25:18.1042604Z"})
- OnNext({"x":"","bp":1308.3,"ap":1308.6,"bs":4.888,"as":57.648,"c":[],"z":"","T":"q","S":"ETH/USD","t":"2022-09-22T12:25:18.1559483Z"})
22/09/2022 14:25:18 Crypto client error: Unable to connect to the remote server
22/09/2022 14:25:19 Crypto client error: Unable to connect to the remote server
22/09/2022 14:25:22 Crypto client error: Unable to connect to the remote server
22/09/2022 14:25:27 Socket opened
22/09/2022 14:25:27 Connected: TooManyConnections
22/09/2022 14:25:30 Socket opened
22/09/2022 14:25:30 Connected: TooManyConnections
22/09/2022 14:25:32 Socket opened
22/09/2022 14:25:32 Connected: TooManyConnections
...
22/09/2022 14:26:57 Socket opened
22/09/2022 14:26:57 Connected: TooManyConnections
22/09/2022 14:27:01 Socket opened
22/09/2022 14:27:01 Connected: TooManyConnections
22/09/2022 14:27:06 Socket opened
22/09/2022 14:27:06 Connected: Authorized
- OnNext({"x":"","bp":19220.0,"ap":19221.0,"bs":0.1625,"as":4.626,"c":[],"z":"","T":"q","S":"BTC/USD","t":"2022-09-22T12:27:09.5580611Z"})
- OnNext({"x":"","bp":19220.0,"ap":19221.0,"bs":0.1625,"as":4.636,"c":[],"z":"","T":"q","S":"BTC/USD","t":"2022-09-22T12:27:09.82687Z"})
- OnNext({"x":"","bp":1309.4,"ap":1309.5,"bs":4.05,"as":28.712,"c":[],"z":"","T":"q","S":"ETH/USD","t":"2022-09-22T12:27:09.858359Z"})
Thanks
Anything else?
No response
@markns I can confirm this behavior for both stocks and crypto connections. Unfortunately, I'm unable to change anything from the client side. But I'll contact with Alpaca back-end team and discuss possible solutions for this problem.
Thanks @OlegRa. Curious to hear what the Alpaca back-end team say. It's a corner case, but would be nice if there was a faster recovery possible.
I re-open it after receiving an update from the Alpaca back-end team. They know about this 90 seconds keep-alive timeout for WS connection but for many reasons they are unable to decrease it. I've proposed to them a solution with an optional boolean flag for the authentication message.
In general, they are OK with the proposed solution but unfortunately, I don't have any estimations from their side about the availability of this new feature for the general audience. As soon as it will be rolled out to the public I'll implement support for it in the SDK and Extensions packages.
@markns After long debates, the server-side guys declined my proposal (and even ready-to-merge code). The main reason for this decision is that "this rarely-used feature adds unnecessary complexity into critical paths of widely used code". I will try to figure out how we can handle these cases better on the client side but I don't have any ETA for now.
Hey @OlegRa, that's too bad, especially after the work you'd put in for the PR. Complexity doesn't come for free though, so I guess we have to respect their decision. As said, it is a corner case, so please feel free to close the ticket wont-fix if you prefer.
@markns In fact, another person prepared the PR with back-end changes, but in any case, we cannot change the current behavior. The WS protocol allows setting up time-to-live intervals for connection via the ping-pong messages exchange. Still, the default .NET ClientWebSocket class only responds to ping messages but never sends them. There are some other WS implementations and the SDK allows us to provide the adapter for them.
I'll not close this issue and try to find an alternative WS client implementation that will work better. I'm not sure how I will redistribute this wrapper in case of success - most probably I'll just post wrapper code here and on the Wiki page.
@markns I've tried several WS client .NET implementations without any success. According to WS standard (RFC 6455), it's not possible to enforce TTL value from the client side. Even if your client sends ping messages every second it means mostly nothing to the server. It can adjust its own timeout value based on these messages but in the case of the Alpaca server, it just sends pong responses. I will try to discuss this problem one more time with the Alpaca back-end guys and ask about decreasing the default TTL down to 30 seconds but I'm unable to promise you anything.