FE/BE - Realtime Data Updates

[ajhollid]

It would be nice if we periodically had new data coming in for a monitor when you are on the details page.

This could be done with:

1. Client side polling
2. Websocket
3. HTTP/2 Server push

Server push would probably be ideal here as we don't need the duplex communication of websocket, this would be strictly one way data tranfer.

Further research into those 3 topics is required for a proposal.

[nishantdecode]

@ajhollid could you please assign me this issue!

[ajhollid]

@ajhollid could you please assign me this issue!

@nishantdecode this is a fairly complex feature that would require a significant rebuild of the front end and the back end. This issue is mostly for research into the options presented above, and any others that you may discover.

If you'd like to do some research and start on a proposal we'd welcome it, but we're not ready to start building this feature yet.

[nishantdecode]

@ajhollid thank you for the guidance! I will do some research and write a proposal.

[gorkem-bwl]

Setting 3.0 milestone for this issue.

[JefferMarcelino]

Hi @ajhollid @gorkem-bwl

I did some research and here’s why the proposal should go with SSE.

HTTP/2 Server push

Let’s start with HTTP/2 Server Push. This option is not ideal and doesn’t solve the problem we want to fix. HTTP/2 Server Push allows the server to proactively send resources to the client before the client requests them. By “resources,” we usually mean static data.

• The server cannot initiate new pushes after the initial request.
• It can only push in response to a client-initiated request.
• After the initial request/response, the server can’t randomly push new data over HTTP/2. The protocol isn’t designed for ongoing unsolicited pushes.

On top of that, major browsers are pulling back support for HTTP/2 Server Push. For example, Google Chrome (the most used browser) has it disabled by default. You can read about it here. Firefox no longer supports it at all, and others are following.

Bottom line: HTTP/2 Server Push is not a viable option here.

Client Side Pooling

Client-side polling means the client sends HTTP GET requests every X seconds to check for new data. It’s simple and fits our current architecture with minimal changes. But it’s not the best approach because:

• Scalability suffers if many clients poll frequently.
• There will always be latency depending on the polling interval.
• Requests will be sent even when there’s no new data.

WebSockets

WebSockets would be ideal for real-time updates but add complexity. We’d need a WebSocket server, and it might be overkill since we only need one-way server-to-client data transfer.

SSE (Server-sent Events)

This wasn’t in your original options, but I strongly believe SSE is the best solution here. Why?

• It’s one-way communication, perfect since we only need server-to-client data. It avoids the extra complexity of bidirectional protocols like WebSockets.
• It has native browser support with the EventSource API.
• The browser automatically reconnects if the network drops.

How SSE works:

• The browser makes a normal HTTP request to a server endpoint (e.g., /my-endpoint).
• The server keeps the HTTP connection open.
• When new data is available, the server sends it as a simple text stream with MIME type text/event-stream.
• The browser automatically handles reconnection if the connection is lost.

Example server data stream:

data: {"cpuUsage": 75}
  
data: {"cpuUsage": 80}

Bottom line: For one-way, real-time data updates on the monitor detail page, SSE offers the best balance of simplicity, browser support, efficiency, and maintainability. It avoids the drawbacks of HTTP/2 Server Push and the complexity of WebSockets while providing real-time updates without constant polling. I recommend moving forward with SSE for this feature.

[gorkem-bwl]

Thanks for the detailed breakdown. That's a screenshot from a discussion on our private channel lately :)

Long story short, we're all on the same page :)