Database change notifications with Server-sent Events (SSE)

[kodeFant]

Ref https://github.com/digitallyinduced/ihp/issues/1800

Usage example: https://github.com/kodeFant/ihp-sse

This is a proposal for enabling EventSouce endpoints that subscribe to database changes through Server-sent Events (SSE), letting you execute custom behaviour on the client triggered by change notifications from the database.

What's the win here, really?

The simplicity of autoRefresh, but allows fine-grained control on what and how updates reflect in the view.

No need to make stateful SPA component islands if you need to only auto-refresh certain parts of the page. Just have the database be the source of truth and sync the data, for example with htmx.

One example that is currently not easy with other techniques in IHP: An always in-sync inbox indicator in the navbar layout showing amount of unread messages.

With autoRefresh, you would need to use autoRefresh everywhere and explicitly track that table in every endpoint since it doesn't take in arguments from the controller, definitely not the right tool for this job.

With DataSync, it's a bit of overkill to set up just for this little feature.

Long-polling could be a viable option, but it means higher latency, more bandwidth-hungry and a higher toll on server resources.

Syncronizing this with SSE is simple, non-hacky, and cheap.

Why SSE

• Simpler that websockets
• Very cheap for both the server and client
• It's nothing fancy, just a part of the web platform and supported by all modern browsers

Browser support: https://caniuse.com/eventsource

Description

Like autoRefresh it tracks tables, but instead of forcing a full page reload, you create an EventSouce endpoint that will subscribe to PostgreSQL event triggers.

Here is the controller code that is needed to make an event listener endpoint. You can track more than only one table if needed:

    action StreamPostsEvents =  withTableReadTracker do
        -- withTableReadTracker will automatically pick up database tables to subscribe to
        trackTableRead "posts"
        streamPgEvent "posts_updated"

Note that even though this is an HTTP endpoint, it does not render html or JSON, but streams an EventSouce subscription to the client.

Practically, this can give you DataSync with equal barrier to entry as autoRefresh.

Some advantages over DataSync is less need for JavaScript, and also very little Haskell code as seen above.

This for example hydrates updated HTML from PostAction into the view each time the posts_updated event is triggered from the StreamPostsEvents endpoint we defined above.

        <div hx-ext="sse" {...[("sse-connect", pathTo StreamPostsEvents)]}>
            <div hx-get={PostsAction} hx-trigger="sse:posts_updated">
                {printPosts posts}
            </div>
        </div>

Instead of HTMX, you could also recieve these events from JavaScript directly by instantiating an EventSource. In other words, this can be equally useful for those who use vanilla JS or jquery.

The example below could be an example solution on how to solve an unread messages indicator as described above.

function initializeEventSource() {
    const eventSource = new EventSource("/StreamPostsEvents");

    eventSource.onopen = function () {
        console.log("Connection opened.");
    };

    eventSource.onerror = function (err) {
        console.error("EventSource failed:", err);
    };

    eventSource.addEventListener('posts_updated', function (e) {
       // Fetch the new count of messages with a dedicated endpoint that will simply respond with a number
        fetch("/PostsCount")
        .then(response => {
            if(!response.ok) {
                throw new Error('Network response was not ok');
            }
            return response.text();
        })
        .then(data => {
            // Update the element with the new message count
            const newMessageCountElement = document.querySelector('#new-message-count');
            if (newMessageCountElement) {
                newMessageCountElement.textContent = data;
            }
        })
        .catch(error => {
            console.error("Failed to fetch the new message count:", error);
        });
    }, false);

    return eventSource;
}

TODO

• [x] Gracefully close heartbeat loop and postgres listener
• [ ] If green light for the implementation, document in IHP guide
• ~~Look at possibility for making SSE endpoints that responds with HTML data directly~~ (I think better to look at later)

[kodeFant]

Open to suggestions about the naming. Not quite sure DBEvent is the coolest, but SSE or EventSource would be too generic, like naming AutoRefresh to Websockets

Edit: Changed to PGEventSource and streamPgEvent, but still open to other suggestions

[s0kil]

At this point, should we consider integrating HTMX as part of IHP?

[kodeFant]

At this point, should we consider integrating HTMX as part of IHP?

Although (just to be clear) this implementation is purely frontend agnostic, I would most certainly use an HTMX boilerplate :)

[kodeFant]

Converting it back to draft for now and letting it sit until I try it in real-life production first. I think the API could be improived

[mpscholten]

Sounds good, let me know when things are for further review 👍

[kodeFant]

@mpscholten do you know if there already exists a good way to use row-level security (or something else) to prevent that the postgres notification triggers each time any column in the tracked tables changes?

[mpscholten]

To make it work with RLS you need to refetch the data on every change. This is how it's done in DataSync as well. You can likely copy the code from here: https://github.com/digitallyinduced/ihp/blob/master/IHP/DataSync/ControllerImpl.hs#L138

It uses sqlQueryWithRLS which you likely want to use here as well (that function just wraps the query so that Row level security policies are applied, so that the refetching is done with RLS enabled)

[kodeFant]

thanks 🙏