Streaming RPC for batches

[jamaljsr]

It would be great to have a streaming RPC endpoint to receive new batches when they are published and/or confirmed on the chain. The events can contain the same data as the current llm auction snapshot response.

[jamaljsr]

Here's some specific events that I would be helpful for the UI.

These are events that occur without any user action, so the only way to get the data is via polling:

• a batch is cleared
• a batch is confirmed on-chain (min confs)
• an order is matched fully or partially (can the event contain the order and the lease)
• an account update is confirmed on-chain (from pending to open/closed)

These are events that occur based on some user action, so data can be refreshed immediately after the action. These are nice to have but not absolutely required:

• an account update is initiated (open, deposit, withdrawal, close)
• an order is updated (submitted, cancelled)

[guggero]

Unfortunately, I didn't get to this yet, partially because of some bugs that needed fixing last week. I have two questions though:

• @jamaljsr I assume you'd like to know the first two events independent of whether the daemon was involved in them or not? To update the list of all known batches?
• If the answer to the above question is yes, then we'd need a streaming RPC on the server side as well that informs about batch updates. To avoid running into resource/connection shortages on our infrastructure, I'd rather avoid creating a second streaming RPC endpoint that requires a long-lived connection to be kept open. Because that would hard limit us to only being able to server 32k traders at once (instead of theoretically 65k traders with just one connection). My proposal would be to just add this new "a batch was just made" message into the existing server stream. The caveat on that is that the stream is only opened after a trader has opened an account. Would that be an UX issue, @justinpobrien?

Alternatively we could just implement the polling in the golang code and push the updates to the UI in a streaming client side RPC. That would mean we probably have to add more caching on the server side to be able to quickly answer those polling calls.

[jamaljsr]

@jamaljsr I assume you'd like to know the first two events independent of whether the daemon was involved in them or not? To update the list of all known batches?

Yes, your assumption is correct. I would like to receive the these events even if the trader was not involved in the batch. We need this info to display all the batches occurring in the auction.

If the answer to the above question is yes, then we'd need a streaming RPC on the server side as well that informs about batch updates. To avoid running into resource/connection shortages on our infrastructure, I'd rather avoid creating a second streaming RPC endpoint that requires a long-lived connection to be kept open. Because that would hard limit us to only being able to server 32k traders at once (instead of theoretically 65k traders with just one connection). My proposal would be to just add this new "a batch was just made" message into the existing server stream. The caveat on that is that the stream is only opened after a trader has opened an account. Would that be an UX issue, @justinpobrien?

I think this would not be good UX as the behavior of the client would differ depending on whether the user has an account or not, and that would not be immediately obvious. I could of course handle these two scenarios differently on the client side. For example, I could do polling when there is no account and streaming when there is. This would require adding a bit of complexity on the UI side but it's not impossible. I would need to know more about when happens when an account is in the pending states or closed while the UI is loaded.

Alternatively we could just implement the polling in the golang code and push the updates to the UI in a streaming client side RPC. That would mean we probably have to add more caching on the server side to be able to quickly answer those polling calls.

I am ok with polling in the golang code if it is more efficient that polling from the browser. This would also eliminate the need for all app devs consuming the poold APIs to implement polling themselves.

[halseth]

@guggero @jamaljsr What is the state of this? Something we still want?

[jamaljsr]

Yes, this is something that would be very useful for the UI as well as others integrating with poold in the future. Currently, the UI is polling, which works but is not ideal.