Streaming endpoints: `byebye` message triggers error

[bartekn]

Try to run a following code:

var server = new StellarSdk.Server({hostname:'horizon-testnet.stellar.org', secure:true, port:443});

var count = 0;
server.accounts()
  .stream({
    onmessage: function(msg) {
      console.log('msg '+(++count));
    },
    onerror: function(err) {
      console.log(err);
    }
  });

The output:

msg 1
msg 2
msg 3
msg 4
msg 5
msg 6
msg 7
msg 8
msg 9
msg 10
{ type: 'error' }
msg 11
msg 12
msg 13
msg 14
msg 15
msg 16
msg 17
msg 18
msg 19
msg 20
{ type: 'error' }
msg 21
msg 22
msg 23
msg 24
msg 25
msg 26
msg 27
...

Every 10 messages connection gets closed (byebye message):

This triggers JS onerror handler. I read the comment about this message but we should probably change this behaviour.

[nullstyle]

I believe this is actually due to us closing the connection on the server side after a query is fulfilled.

I'd love to find some time to investigate how to best handle this scenario, but I don't think it would be prudent for us to enable unbounded streaming... at present our rate-limiting architecture only operates at the http request level and allowing unbounded streams would expose us to capacity issues, IMO.

If you want to investigate how to cleanly close the event source connection, feel free. I wasn't able to find much documentation about how to do it. At the very least, we should wrap the EventSource object we on the client side such that we can hide that error from people using js-stellar-sdk

[bartekn]

OK, here's what I found out so far:

• Only node eventsource does reconnect after we close a connection. This means that (if we won't find other solution than closing a connection) we should reconnect in browsers in stellar-sdk.
• Ignoring this error in JS libs is not possible right now (at least with eventsource package). eventsource throws generic Error object and we can't read a message sent with that event.

Possible solutions:

• Fork eventsource (for node) and event-source-polyfill (for IE) and make them a) reconnect after closing a connection b) not to trigger onerror event when this happens.
• Wondering if it's possible: can we call RateLimit directly and close eventsource connection when we detect that rate limit exceeded?

[bartekn]

@nullstyle you were right. Browsers do reconnect. I was testing this in our network explorer and it executes: es.close() in onError handler - that's why it was stopping after the first page of results.

[elucidsoft]

Testing this with .NET Core HttpClient I seem to be getting an error. I am not sure if its a problem on the .NET side or a problem with how the SSE chunking is working. I have posted an issue to the .NET Core team to take a look. If I inspect a stream via Fiddler for instance it tells me your stream is invalid because it is not terminating properly with a 0 sized chunk. I am not sure if this is related to the 'byebye' or not.