[🚀 Feature]: [BIDI] Add commands to retrieve req/rsp bodies

[newsgrep]

Description

Feature and motivation Selenium already has some Network Intercept BiDi APi which is given in the example of BiDi API. However, this API can only be used to deal with some meta-data and headers. At the moment I must rely on CDP to inspect the payload.

I would be grateful and it would help me a great deal to have a true cross-browser implementation that makes use of the WebDriver BiDi spec that can now capture the request/response with header and body.

Usage example Implement something similar to getUrl() or getHeaders(); but to get the payload of a request / response. e.g. #java

// Listener für Request & Response:
network.onBeforeRequestSent(request -> {
    System.out.println("REQUEST URL: " + request.getRequest().getUrl());
    System.out.println("REQUEST HEADERS: " + request.getRequest().getHeaders());
    System.out.println("REQUEST BODY: " + request.getRequest().getBody());
});
network.onResponseStarted(response -> {
    System.out.println("RESPONSE URL: " + response.getResponseData().getUrl());
    System.out.println("RESPONSE HEADERS: " + response.getResponseData().getHeaders());
    System.out.println("RESPONSE BODY: " + response.getResponseData().getBody());
});

It is now supported in:

• the sepc: network.addDataCollector and network.getData
• Firefox 143:
• • https://bugzilla.mozilla.org/show_bug.cgi?id=1971780
• • https://bugzilla.mozilla.org/show_bug.cgi?id=1971778
• Chrome: https://github.com/GoogleChromeLabs/chromium-bidi/blob/main/src/bidiMapper/modules/network/NetworkStorage.ts#L518

More information https://github.com/SeleniumHQ/selenium/issues/16261 https://github.com/SeleniumHQ/selenium/issues/11872 https://github.com/w3c/webdriver-bidi/issues/958 https://github.com/w3c/webdriver-bidi/pull/877 https://github.com/w3c/webdriver-bidi/pull/1011 fyi: @TamsilAmani @titusfortner

Have you considered any alternatives or workarounds?

I tried to use devTools.send and devTools.addListener but was not able to get it to work in Firefox (only Chrome), using BIDI to get the url in both worked but I'm looking for a cross-browser way of inspecting the body of request/responses.

[selenium-ci]

@newsgrep, thank you for creating this issue. We will troubleshoot it as soon as we can.

Selenium Triage Team: remember to follow the Triage Guide

[navin772]

Hi @newsgrep, there are already open PRs for implementing this feature across Python and Java. Dotnet already supports this (PR).

1. Python - https://github.com/SeleniumHQ/selenium/pull/16438
2. Java - https://github.com/SeleniumHQ/selenium/pull/16336

[newsgrep]

Intreasting, thank you. The way I read the spec it defines a request and response.

network.DataType = "request" / "response" The network.DataType type represents the different types of network data that can be collected. https://www.w3.org/TR/webdriver-bidi/#type-network-dataType

But in both PRs I only see data_types=["response"] and DataType.RESPONSE, how can I read the body of the requests, e.g. for a POST?

[navin772]

But in both PRs I only see data_types=["response"] and DataType.RESPONSE

The request DataType was added recently in the bidi spec - https://github.com/w3c/webdriver-bidi/pull/1011 and the Java PR was created before that, Python doesn't limits you to use only response. I will update the Java PR to add an ENUM entry for request.

But do note that stable browsers as of now don't support request data type, so you will still see errors like - invalid argument: Expected "dataTypes" values to be one of response, got [object String] "request".

[newsgrep]

Thank you very much for pointing that out. I was not aware of that.

Seems to be tracked for Firefox, did not find it for Chrome. 1988955 - Add support for dataType "request" to addDataCollector, getData and disownData

[newsgrep]

It looks like Chrome may support the request data type already: browser-compat-data/webdriver/bidi/network.json at 6d3a256afac3a8f6c383d706bfe3d8f53bc6391e · mdn/browser-compat-data · GitHub feat: support request post data collection by sadym-chromium · Pull Request #3815 · GoogleChromeLabs/chromium-bidi · GitHub

[navin772]

It looks like Chrome may support the request data type already:

Its not supported on Chrome 142 which is current stable version, but you can check on Beta/Canary versions, I think it should be supported in Canary.

[newsgrep]

will update the Java PR to add an ENUM entry for request.

@navin772 Firefox 146 is stable now and does support requests, it would be much appreciated if you could update the PR and check if it works.

From the Firefox changelog:

Updated all our network data collection commands (network.addDataCollector, network.getData and network.disownData) to support the request data type, which allows to collect and retrieve request post data. (Firefox bug 1988955).

Improved our implementation for network.getData to also support requests using the data: scheme. (Firefox bug 1992210).

[navin772]

@navin772 Firefox 146 is stable now and does support requests, it would be much appreciated if you could update the PR and check if it works.

@newsgrep I have tested with python, request type is now working in all 3 browsers. I will update the Java PR soon!