Support mocking WebSocket APIs

[Doesntmeananything]

Is it possible to use msw to mock server-sent events (SSE) and WebSocket (WS) connections?

My use case is to have a somewhat full client-side mocking story (including REST, SSE, and WS functionality), and since msw is such a joy to use when mocking out REST APIs, I was wondering if it makes sense to use it to mock more specialised server interactions.

Have you thought about this? I admit that I haven't looked much into whether it's possible just to use custom request handlers to add this functionality, emulating SSE and WS behaviour in some way. I wanted to know if you already had something in mind regarding this question. Thanks!

[kettanaito]

Hey, @Doesntmeananything, thanks for bringing this topic up. I'd love to bring SSE and WS support to the users of MSW. I admit I haven't researched the topic yet, but would use this thread for this.

Technically, it comes down to the ability of Service Worker to intercept those requests/events. If the spec supports it, there shouldn't be much changes needed on the MSW side.

Here's some useful resources:

• https://github.com/w3c/ServiceWorker/issues/885, EventSource interception in Service Worker.
• https://github.com/w3c/ServiceWorker/issues/947, a discussion whether Service Worker should access WebSocket events.

Could you please try to set up a proof of concept, if those events can be intercepted in the worker's fetch event?

You're right about the custom request handler, we can use it to log all intercepted requests:

setupWorker(
  {
    predicate(req) {
      console.log(req)
      // log captured requests, but bypass them
      return false
    },
    resolver: () => null
  }
)

If we confirm it working, I'd be glad to discuss the API for a dedicated request handler for WebSocket/SSE. I can read on their specification meanwhile.

[Doesntmeananything]

Sounds like a plan! At a cursory glance, it does indeed seem quite doable. Let me PoC this, and I'll get back to you with my results as soon as I can.

[Doesntmeananything]

Hi, @kettanaito! I've set up a (very quick and dirty) repository to test these interactions over at https://github.com/Doesntmeananything/msw-sse-ws.

My initial findings are the following:

• I am able to intercept SSE connections, however this occurs only when a stream ends. For example, if a stream consists of 3 messages, logging happens only after all 3 messages have been received, including the final end event.
• I am not able to intercept WS connections. I tried to make sure that the WS connection is established from the client only after the mock service worker had been initialised, but it didn't seem to help. I want to investigate this further.

I'm a bit concerned about WS events, although I hope that with some additional work it'd possible to intercept them.

[kettanaito]

@Doesntmeananything, thank you for the investigation! I'm excited to hear that SSE can be intercepted! Wonder if there's anything we can do it intercept events as they go.

I'm currently working on a NodeJS support, but can switch to this issue to help you once I'm done. I'm always open to questions or discussions, so please don't hesitate to raise those here.

Also, if you don't mind, we could then move your proof of concept repo under "msw" to serve as an example how to work with SSE/WS. That'd be awesome.

[kettanaito]

I'm trying to get my head around the SSE example. It seems MSW should intercept the hi from client event, so then it can mock the server response to it. I can see the once the WS connection is established, all the messages are inspectable live in DevTools. However, the webSocket.send("hi from client") is not intercepted by the Service Worker. I'm reading through https://github.com/w3c/ServiceWorker/issues/947, trying to figure out if it's technically possible to access WS communication in a service worker.

API-wise, I think there should be at least two types of request handlers: event-based handler, and persistent handler (pulsing back messages to the client, like you have in your example using AsyncIterator).

[kettanaito]

One of the most useful pieces of code I've found in the w3c discussion (https://github.com/w3c/ServiceWorker/issues/947#issuecomment-410816076) was that the Service Worker file can establish a WebSocket connection. It appears that the WS events are not subjected to be intercepted in the fetch event, but one can establish a socket connection an (?) intercept events that way.

If it comes down to the manual WS connection, I'd suggest to do that on the client's side, not in the worker file. There's no technical reason to move this logic to worker, at least as of how I understand such implementation now.

[Doesntmeananything]

Thanks very much for taking the time to look further into this!

Since I've hit the wall in regards to intercepting WS connections, your suggestions come in handy. Will definitely look into this.

To be clear, are you saying that mocking WS connections falls strictly outside of MSW concerns? My investigations lead me to believe this, and I would certainly not want to push for something that doesn't make sense neither on technical nor on conceptual level.

[kettanaito]

Not necessarily. What I was trying to say is that a WebSocket event is not intercepted by the fetch event in a Service Worker. That's a per-spec behavior. However, I've mentioned an example above, that creates a WS connection within the worker file, which I suppose allows to intervene the communication in some way. I haven't tried that approach out, whether it's actually possible to mock the response of an event.

[kettanaito]

I've received a suggestion to look at mock-socket. We may get some inspiration from how it's implemented, and see if a similar approach can be done in MSW.

[kettanaito]

Update: I've started with the WebSocket support and will keep you updated as I progress. For those interested I will post some technical insights into what that support means, what technical challenges I've faced, and what API to expect as the result.

Session 1: It's all about sockets

No service for the worker

Unfortunately, WebSocket events cannot be intercepted in the fetch event of the Service Worker. That is an intentional limitation and there's no way to circumvent it. This means a few things:

• WebSocket events won't be visible in the "Network" tab;
• WebSocket support does not require the worker and can live outside of setupWorker context.
• WebSocket events interception should be done by patching the underlying logic (i.e. a WebSocket class).

Goodbye, handlers!

WebSocket operates with events, not requests, making the concept of request handler in this context redundant. Instead, you should be able to receive and send messages from ws anywhere in your app, including your mock definition.

import { rest, ws, setupWorker } from 'msw'

// Create an interception "server" at the given URL.
const todos = ws.link('wss://api.github.com/todos')

setupWorker(
  rest.put('/todo', (req, res, ctx) => {
    const nextTodos = prevTodos.concat(req.body)
    
    // Send the data to all WebSocket clients,
    // for example from within a request handler.
    todos.send(nextTodos)
    return res(ctx.json(nextTodos))
  })
)

// Or as a part of arbitrary logic.
setInterval(() => todos.send(Math.random()), 5000)

URL that got away

When constructing a WebSocket instance you must provide a URL that points to a WebSocket server. Providing a URL to a non-existing server yields and exception that you cannot circumvent by patching WebSocket class, as the URL validation lives in its constructor.

I've ended up re-implementing a WebSocket class, effectively making a polyfill out of it. That way it can have its custom constructor that would tolerate non-existing URLs if there is a ws.link interception server declared for that URL.

Persisting WebSocket clients

The entire idea of WebSocket is to sync data between multiple clients in real time. When you dispatch a mocked ws.send event to send some data to all clients, you need to let all the clients know they should receive the data (trigger their message event listener). However, there's no way to know and persist a list of WebSocket clients on runtime, since each page has its own runtime.

Usually a solution to this kind of problems is to lift the state up and maintain a record of clients in the upper context shared with all the clients (pages). However, in JavaScript there isn't that may ways to share and persist data between clients. In case of WebSocket clients one needs to store references to WebSocket instances—basically, object references. I've considered:

• localStorage/sessionStorage. Good for sharing textual data, not that suitable for storing objects with prototypes. Objects flushed here effectively lose their references, making them completely different objects.
• Web Worker/Service Worker. A great way to have a detached process in a browser that can control and communicate with multiple pages. However, as stated before, worker API cannot intercept WebSocket events, and using it only for the sake of storing some records is utterly redundant (I don't wish to ask people to copy any more worker scripts). Utilizing an existing mock worker may be an option, however, I'm afraid it would put too many logic into it, increasing its maintenance, and violating its purpose of containing a bare minimum logic that you seldom need to update.
• BroadcastChannel. Turns out the API that allows workers to communicate with clients exists standalone and it's awesome. You can create a broadcast channel as a part of page's runtime, and as long as another page on the same host creates a channel with the same name they can send data between them.

const channel = new BroadcastChannel('ws-support')

// One client sends a data.
channel.send('some-data')

// All clients can react to it.
channel.addEventListener('message', (event) => {
  event.data // "some-data"
})

I find BroadcastChannel a great choice to mimic the real time data synchronization functionality of WebSocket. I've chosen it to spawn a single channel between all clients and notify them when they should trigger their message event listeners.

[Sun-2]

@kettanaito

URL that got away When constructing a WebSocket instance you must provide a URL that points to a WebSocket server. Providing a URL to a non-existing server yields and exception that you cannot circumvent by patching WebSocket class, as the URL validation lives in its constructor. I've ended up re-implementing a WebSocket class, effectively making a polyfill out of it. That way it can have its custom constructor that would tolerate non-existing URLs if there is a ws.link interception server declared for that URL.

You could use an ES6 Proxy. It can mess with ctors.

Link.

[BlackGlory]

SSE and WebSockets are different issues. If msw supports response streams (such as ReadableStream), it can support SSE.

[kettanaito]

@BlackGlory, MSW should support ReadableStream as the mocked response body. Would you have some time to try to which extent that's true, and whether SSE would be supported now?

[BlackGlory]

@kettanaito Although ctx.body supports ReadableStream, it does not seem to work.

export const worker = setupWorker(
  rest.get('/sse', (req, res, ctx) => {
    return res(
      ctx.status(200)
    , ctx.set('Content-Type', 'text/event-stream')
    , ctx.body(sse(function* () {
        yield 'message1'
        yield 'message2'
      }))
    )
  })
)

function sse(gfn) {
  let iter
  return new ReadableStream({
    start() {
      iter = gfn()
    }
  , pull(controller) {
      controller.enqueue(`data: ${iter.next().value}\n\n`)
    }
  })
}

Browser:

[MSW] Request handler function for "GET http://localhost:8080/sse" has thrown the following exception:

DOMException: Failed to execute 'postMessage' on 'MessagePort': ReadableStream object could not be cloned.
(see more detailed error stack trace in the mocked response body)

Node.js:

TypeError: The first argument must be of type string or an instance of Buffer, ArrayBuffer, or Array or an Array-like Object. Received an instance of ReadableStreamTypeError [ERR_INVALID_ARG_TYPE]: The first argument must be of type string or an instance of Buffer, ArrayBuffer, or Array or an Array-like Object. Received an instance of ReadableStream

      at ClientRequestOverride.<anonymous> (node_modules/node-request-interceptor/src/interceptors/ClientRequest/ClientRequestOverride.ts:216:34)
      at step (node_modules/node-request-interceptor/lib/interceptors/ClientRequest/ClientRequestOverride.js:33:23)
      at Object.next (node_modules/node-request-interceptor/lib/interceptors/ClientRequest/ClientRequestOverride.js:14:53)
      at fulfilled (node_modules/node-request-interceptor/lib/interceptors/ClientRequest/ClientRequestOverride.js:5:58)

[kettanaito]

Hey, @BlackGlory. Could you please report this as a separate issue? Thanks.

[MarkLyck]

any update on web socket support? 🙏

It would be amazing to get this built into msw.

I'm currently using MSW for all normal requests, but I have yet to find any good solution for mocking my 1 web socket service which is causing me grief.

[kettanaito]

Hey, @MarkLyck. The support is in progress, but we're prioritizing some more pressing bug fixes at the moment.

You can follow the progress here #396. The in-browser support looks good, needs polishing. The NodeJS support is on the roadmap. I have some concerns over NodeJS usage, as WebSocket is not a part of the standard library (to my knowledge). I'd rather not build any client-specific logic around web sockets.

[kettanaito]

Update

The proof of concept for WebSocket support in a browser is functional. As mentioned earlier, it can be achieved by patching the native WebSocket class that's a standard browser API.

The Node.js support remains unknown, as there's no native WebSocket class to operate with. We'd have to reverse-engineer how most common WebSocket libraries function in Node.js (i.e. socket.io) and see what kind of requests they create if any. The end goal is to provide functionality that's library-agnostic and works no matter what WebSocket client you've decided to use. So far there's no confirmation it'll be the case in Node.js.

When to expect this?

While WebSocket support is a great feature to have, we prioritize other features that are objectively more useful to the majority of our users. There's no deadline on WebSocket support, as it's approached in a free time when there are no other pressing issues or features to address.

How to make this happen?

We'd be thankful for the investigation on how WebSocket libraries function in Node.js, specifically:

• What kind of requests do they make? Are those instances of ClientRequest or something else?
• How can a WebSocket event be intercepted in Node.js? At which level a patch to WebSocket should be introduced?

[jbcpollak]

Hi @kettanaito thanks for the update - do you think you can release support sse in browsers or do you need to wait till there is parity with nodejs? If you'd release browser support, what do you need to get that done?

[kettanaito]

Hey, @jbcpollak. We will release this feature only when it's fully functional in both browser and Node.js.

As I've mentioned above, the initial phase of the browser support is completed, and you can install it in your projects:

$ npm install mswjs/msw#pull/396/head
$ yarn add mswjs/msw#396/head

Although looking at the build status on that pull request, it doesn't pass, so you're unlikely to use it this way, sorry. You can build it locally from that pull request, I verify that npm run build command is okay.

We'd be thankful if you tried it out and provided us with your feedback!

[jbcpollak]

Hi, I gave it a go using the npm link method and added some comments about problems on the PR. Only after getting it working I realized the PR is for web sockets, and I need Server Sent Events support, so I'm stuck for now.

[kettanaito]

SSE in Service Worker

I've found a great article that goes into detail on how to intercept and respond to SSE using Service Worker:

• Server-Sent Events Gateway by Service Worker

This looks like a good starting point to explore to enable SSE support.

Note that WebSocket events still can't be intercepted. At least I couldn't achieve that during the development. If that's the case, and SSE and WS should be treated differently, then I'd suggest splitting this issue into two phases: SSE and WS support respectively.

[dbehmoaras]

@kettanaito this would be very helpful indeed. It seems that the current behavior handles a websocket interception (via socket.io) by throwing a warning. Until the new library is released, do you have a suggestion for how to handle and/or disable this warning?

[lukas-mertens]

@kettanaito Thank you for experimenting on this. I haven't used mswjs yet and don't know if this possibly covered by some other feature, but I wanted to add my usecase of websocket-support anyways, as I believe this issue has to be resolved to achieve what I am trying to do:

According to the GraphQL specification, section 6.2.3 graphql supports subscriptions. This allows the server to push updates to query-results to the client. It would be very nice, if it was possible to mock this behavior as well. If this is already possible and I just missed something in the docs, just ignore what I said.

[kettanaito]

Hey, @lukas-mertens. Yes, you're right in thinking that GraphQL subscription mocking will benefit greatly from WebSocket mocking support (hence #285). This pull request focuses on mocking a WebSocket server, which currently functions as expected in the browser. Once we release this feature (if ever), we'll follow up with the GraphQL subscriptions support, letting you push updates from the server.

If you or your team can use a unified way to mock WebSocket and subscriptions across platforms, consider supporting the effort of our team. Any contribution brings features like this closer to reality. Thanks.

[mariothedev]

I was able to successfully mock a SSE!!!

Here's how:

1. Your mocked endpoint must have the following properties:

export const sseMock = rest.get(
    `http://localhost:3000/stream`,
    (_, res, ctx) => {
        console.log('mock SSE server hit')
        return res(
            ctx.status(200),
            ctx.set('Connection', 'keep-alive'),
            ctx.set('Content-Type', 'text/event-stream'),
            ctx.body(`data: SUCCESS\n\n`)
        )
    }
)

2. Comment out the following from the mockServiceWorker.js file:

// Bypass server-sent events.
    // if (accept.includes('text/event-stream')) {
    //     return
    // }

And that's it! 👍

After that just insert your mocked response under setupWorker:

export const worker = setupWorker( someOtherMockResponse, sseMock )

[SerkanSipahi]

@mariothedev does your suggestion any negative impact to the usage of msw?

[mariothedev]

@SerkanSipahi - not really. I've been using like this for quite a while now and no issues as of yet.

[dbehmoaras]

I was able to successfully mock a SSE!!!

This is a really neat solution!

[kettanaito]

Update: I'm back on the WebSocket support. Currently, I've implemented the native window.WebSocket interceptor in @mswjs/interceptors and working/testing the interception of SocketIO in particular.

Meanwhile, I'd like to toss around some thoughts on the ws API in MSW. I'd very much like everybody following this feature to contribute their thoughts and ideas around that API.

The ws API

First of all, just like @mswjs/data models its API around Prisma because it's a superb querying client, I think the ws API should model itself around SocketIO. SocketIO is de-facto a standard for WebSocket implementations and is widely used.

Usage example

Here's an extensive usage example of a hypothetical ws API in MSW:

import { ws } from 'msw'

export const handlers = [
  ws.link('wss://api.mychat.com', (io) => {
    // Handles when a new socket connects.
    io.on('connection', (socket) => {
      // Handles the "message" events from this socket.
      socket.on('message', handler)

      // Handles a custom event from this socket.
      socket.on('greet', (username) => {
        // Sends data to this socket.
        socket.send('hello to you too', username)

        // Send data to other sockets but this one.
        socket.broadcast.emit('hello everybody else!')
        socket.broadcast.send('sends to everybody')
      })
    })
    
    // Sends data to all connected sockets.
    io.send('hello')
    io.emit('greet', 'John')

    // Handles when a socket disconnects.
    io.on('disconnect', (socket) => {
      console.log('socket %s disconnected', socket.id)
    })
  })
]

General statements

• Mocking a WebSocket API would require you to specify the WebSocket server endpoint via ws.link(). Intercepting any events to any WebSocket servers is not what you want to do anyway, as that pushes the endpoint distinguishing logic on your shoulders.

Capabilities

Here's what I think the ws API should enable our users:

1. Listen when a new client connects: io.on('connection').
2. Listen when a client disconnects: io.on('disconnect').
3. Send a message to all clients: io.send().
4. Send a message to the currently connected client: socket.send().
5. (SocketIO only) Emit a custom event to all clients: io.emit().
6. (SocketIO only) Send a custom event to the currently connected client: socket.emit().
7. (SocketIO only) Send/emit data to all clients except the currently connected one: socket.broadcast.send()/socket.broadcast.emit().

As SocketIO introduces custom features (such as socket.emit and socket.broadcast), I feel that MSW should enable those features in the mocks. We have to be cautious not to end up re-implementing SocketIO internally though, so I'd expect certain SocketIO features not to be supported at least in the initial ws release.

Question to you

• What do you think about the overall ws API look and feel above?
• Are there any scenarios that this API couldn't handle? Please share them below.
• Is there anything missing from this API to mock your real-world WebSocket applications?
• Should you be able to spy on the events sent from the actual running server? To have an introspection of sorts.