[network-plugin] Feat: Capture network events

[jlalmes]

Closes https://github.com/rrweb-io/rrweb/issues/552

This implementation records fetch and XMLHttpRequest by patching their object & methods. We record document navigation using PerformanceNavigationTiming and we use PerformanceResourceTiming for recording everything else (script, img, link etc.) via PerformanceObserver API.

Recording

type NetworkRequest = {
  url: string;
  method?: string;
  initiatorType: InitiatorType;
  status?: number;
  startTime: number;
  endTime: number;
  requestHeaders?: Headers;
  requestBody?: string | null;
  responseHeaders?: Headers;
  responseBody?: string | null;
};

rrweb.record({
  ...,
  plugins: [
    getRecordNetworkPlugin({
      initiatorTypes: ['fetch', 'xmlhttprequest'], // omit to record all types
      ignoreRequestFn: (request: NetworkRequest) => false
      recordHeaders: true, // defaults to false
      recordBody: true, // defaults to false
      recordInitialRequests: true, // defaults to false
    })
  ]
})

• initiatorTypes is an array specifying which request types to record
• ignoreRequestFn is a function that allow you to block recording an event for a request (you don't want to create a loop recording the request you send to store rrweb events on your server)
• recordHeaders will record an request/response headers for fetch and xmlhttprequest
• recordBody will record an request/response bodies for fetch and xmlhttprequest
• recordInitialRequests will record an event for all requests prior to rrweb.record() being called

Replaying

new rrweb.Replayer(events, {
  ...,
  plugins: [
    getReplayNetworkPlugin({
      onNetworkData: (data: { requests: NetworkRequest[], isInitial?: boolean }) => { // required
        // do something
      }, 
    })
  ]
})

• You must supply your own onNetworkData function.

To-do

• Tests

[jlalmes]

Thanks for your review @Mark-Fenng - I have just updated this branch with all of the changes you have suggested! I plan to add some tests shortly.

Out of interest do you know when we plan to release 2.0.0-alpha.5?

[YunFeng0817]

Thanks for applying all review suggestions. Although there are still a few small points that need to tweak, it's very close to the final version.

FYI here is a TODO list:

• [x] improve the content type filter
• [ ] test cases
• [ ] Chinese docs (I will add them later)

I'm so sorry that, for the release of 2.0.0-alpha.5, I also don't have a specific date to tell. These days other team members are all busy with their work and some important pull requests for bug fixes are delayed.

[Juice10]

This looks great! I agree with @Mark-Fenng, if we have some more tests this will be ready to merge. Also I'm happy to spend some time working on the 2.0.0-alpha.5 release soon

[changeset-bot[bot]]

⚠️ No Changeset found

Latest commit: 8863576a1abb51366c0156d8d8b64c9656baf0ee

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[benjackwhite]

@jlalmes need some help getting this across the finish line? This would be such a good improvement to the custom solution we have for network recording and things here look pretty solid, just needing some good tests it seems.

[yaoyuanrong]

Is this plugin released?

[yaoyuanrong]

I really need this feature。I think this is a very powerful feature @jlalmes

[jasonfill]

I would love this feature! Just throwing in a 👍 for this.

[pathik-appachhi]

Hello,

I have recently started using rrrweb-io via the cdn . It would be very helpful to know when this plugin will be available on the CDN link.

[ShayMalchi]

@jlalmes @YunFeng0817 Seems like this PR was approved long ago, can you please share what is the current blocker for merging? Many people in this thread are interested in it, and seems like most suggestions could be handled in followup PRs. Thank you!

[pauldambra]

I'll just re-up this comment https://github.com/rrweb-io/rrweb/pull/1105#discussion_r1402360817

It's easier for me if we have name not URL. Since the performance observer uses name and emits more than just things with URLs.

[pauldambra]

We have network payload capture in prod now based on this PR - thanks so much @jlalmes 🙌

You can see the code here https://github.com/PostHog/posthog-js/blob/main/src/loader-recorder-v2.ts

We couldn't use it quite as it is... and I've had to make a couple of bugfixes since releasing it - mostly around processing bodies on unexpected types of network requests

My plan is to let it settle in our product for a week or two. And then I'll open a PR based on this one that would let us inject the things we'd varied - which might also be things other people would want to vary...

It won't be massively different - and of course the version that works for us might not be preferable generally

[sharan-ravi]

@jlalmes @pauldambra Thank you so much for this!

I was running into an issue with some of my requests not being recorded with this plugin, after digging deeper, turns out in case of certain requests, if the server doesn't return a Timing-Allow-Origin response header, we wouldn't be able to track the performance metrics. Given this plugin throws an error when performance metrics are not available, these requests were not being recorded.

While the performance metrics are not available, Other data on these requests are still available, so in my case I had to do something like this to still record these requests.

getRequestPerformanceEntry(win, 'fetch', req.url, after, before)
                    .then((entry) => {
                        if (_isNull(entry)) {
                            const requests = prepareRequestWithoutPerformance(req, networkRequest)
                            cb({ requests })
                        } else {
                            const requests = prepareRequest(entry, req.method, res?.status, networkRequest)
                            cb({ requests })
                        } })

function prepareRequestWithoutPerformance(
    req: Request,
    networkRequest: Partial<CapturedNetworkRequest>,
): CapturedNetworkRequest[] {

    const reqJson = {
        url: req.url,
        method: req.method,
        requestHeaders: networkRequest.requestHeaders,
        requestBody: networkRequest.requestBody,
        responseHeaders: networkRequest.responseHeaders,
        responseBody: networkRequest.responseBody,
    }

    return [reqJson]
}

Is this expected?

[Juice10]

My plan is to let it settle in our product for a week or two. And then I'll open a PR based on this one that would let us inject the things we'd varied - which might also be things other people would want to vary...

It won't be massively different - and of course the version that works for us might not be preferable generally

@pauldambra would you still be interested to open a PR based on the version you have running in production?

FYI: https://github.com/rrweb-io/rrweb/pull/1033 just got finished which moves the all of the plugins to the packages/plugins/... directory. Might be worth considering creating the PR based on that as well

[akhsri]

Hi @jlalmes , When can we see this in rrweb package? it would be very powerful and helpful feature!

[pauldambra]

@Juice10 definitely... our implementation has been pretty stable for a while now... it's just time that's stopping me 🙈

I think the "tricky" thing will be separating our config/mapping needs out of the plugin but that's pretty mechanical non-ground-breaking stuff

[Marlonzao]

Hello, is there a guide on how to download this plugin?

[Marlonzao]

I have figuret it, you have to clone this repo https://github.com/jlalmes/rrweb/tree/feat/network-plugin then ckeckout to the branch "feat/network-plugin" and run "yarn && yarn run build:all" the files needed will be in the directory packages/rrweb/dist