[🐛 Bug]: A command response was not received: Network.getResponseBody C#

[lopukhDA]

What happened?

Hello, I am constantly getting an error when I try to get the response body.

One or more errors occurred. (A command response was not received: Network.getResponseBody)

StackTrace:

  at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at ConsoleApp3.Program.<>c__DisplayClass0_0.<Main>g__RequestWillBeSentHandler|0(Object sender, RequestWillBeSentEventArgs e) in C:\Users\ABC\source\repos\ConsoleApp3\ConsoleApp3\Program.cs:line 34

How can we reproduce the issue?

static async Task Main(string[] args)
        {
            var options = new ChromeOptions();

            var driver = new ChromeDriver(ChromeDriverService.CreateDefaultService(), options, TimeSpan.FromMinutes(2));

            var devTools = (IDevTools)driver;
            var session = devTools.GetDevToolsSession();
            var domains = session.GetVersionSpecificDomains<DevToolsSessionDomains>();

            void RequestWillBeSentHandler(object sender, RequestWillBeSentEventArgs e)
            {
                if (e.Request.Url.Contains("google.com"))
                {
                    try
                    {
                        var getResponseBody = domains.Network.GetResponseBody(new GetResponseBodyCommandSettings()
                        {
                            RequestId = e.RequestId
                        });

                        var body = getResponseBody.Result.Body;
                    }
                    catch (Exception ex)
                    {
                        Console.WriteLine(ex);
                    }
                }
            }

            domains.Network.RequestWillBeSent += RequestWillBeSentHandler;
            await domains.Network.Enable(new EnableCommandSettings());

            driver.Navigate().GoToUrl("https://www.google.com/");

            driver.Quit();
        }

Relevant log output

Starting ChromeDriver 96.0.4664.45 (76e4c1bb2ab4671b8beba3444e61c0f17584b2fc-refs/branch-heads/4664@{#947}) on port 55559
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.

DevTools listening on ws://127.0.0.1:55562/devtools/browser/7a626fcd-3684-445b-9b6f-ebdb992b3712
[29984:27600:1214/114346.774:ERROR:chrome_browser_main_extra_parts_metrics.cc(226)] crbug.com/1216328: Checking Bluetooth availability started. Please report if there is no report that this ends.
[29984:27600:1214/114346.774:ERROR:chrome_browser_main_extra_parts_metrics.cc(229)] crbug.com/1216328: Checking Bluetooth availability ended.
[29984:27600:1214/114346.774:ERROR:chrome_browser_main_extra_parts_metrics.cc(232)] crbug.com/1216328: Checking default browser status started. Please report if there is no report that this ends.
[29984:27600:1214/114346.780:ERROR:chrome_browser_main_extra_parts_metrics.cc(236)] crbug.com/1216328: Checking default browser status ended.
System.AggregateException: One or more errors occurred. (A command response was not received: Network.getResponseBody)
 ---> System.InvalidOperationException: A command response was not received: Network.getResponseBody
   at OpenQA.Selenium.DevTools.DevToolsSession.SendCommand(String commandName, JToken commandParameters, CancellationToken cancellationToken, Nullable`1 millisecondsTimeout, Boolean throwExceptionIfResponseNotReceived)
   at OpenQA.Selenium.DevTools.DevToolsSession.SendCommand[TCommand,TCommandResponse](TCommand command, CancellationToken cancellationToken, Nullable`1 millisecondsTimeout, Boolean throwExceptionIfResponseNotReceived)
   at OpenQA.Selenium.DevTools.V96.Network.NetworkAdapter.GetResponseBody(GetResponseBodyCommandSettings command, CancellationToken cancellationToken, Nullable`1 millisecondsTimeout, Boolean throwExceptionIfResponseNotReceived)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at ConsoleApp3.Program.<>c__DisplayClass0_0.<Main>g__RequestWillBeSentHandler|0(Object sender, RequestWillBeSentEventArgs e) in C:\Users\ABC\source\repos\ConsoleApp3\ConsoleApp3\Program.cs:line 34

C:\Users\ABC\source\repos\ConsoleApp3\ConsoleApp3\bin\Debug\net5.0\ConsoleApp3.exe (process 14516) exited with code 0.
To automatically close the console when debugging stops, enable Tools->Options->Debugging->Automatically close the console when debugging stops.
Press any key to close this window . . .

Operating System

Windows 10

Selenium version

Selenium 4.1, .Net Core 5 or .NET Framework 4.7.2(C# 8)

What are the browser(s) and version(s) where you see this issue?

Chrome 96

What are the browser driver(s) and version(s) where you see this issue?

ChromeDriver 96

Are you using Selenium Grid?

no

[github-actions[bot]]

@lopukhDA, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[amather]

I'm having the same issue, but running the CDP command directly works for me:

Dictionary<string, object> cdpParameters = new(){ { "requestId", e.RequestId} };
Dictionary<string, object> res = (Dictionary<string, object>) ((ChromeDriver)drv).ExecuteCdpCommand("Network.getResponseBody", cdpParameters);
 if (res.TryGetValue("body", out object? bodyObj))
 {
    string body = (string)bodyObj;
 }

[Dreamescaper]

@lopukhDA You're trying to retrieve response body on RequestWillBeSent event, response body not available at this point, since there is no response itself so far. You should do that on LoadingFinished event.

[cgv-wd]

I'm having the same issue, but running the CDP command directly works for me:

Dictionary<string, object> cdpParameters = new(){ { "requestId", e.RequestId} };
Dictionary<string, object> res = (Dictionary<string, object>) ((ChromeDriver)drv).ExecuteCdpCommand("Network.getResponseBody", cdpParameters);
 if (res.TryGetValue("body", out object? bodyObj))
 {
    string body = (string)bodyObj;
 }

Just want to second what @amather mentioned. I had trouble implementing the usage of the new auto generated command classes for version specific domains. I may have been misusing them, but it would be nice if there was a document with examples on recommended usage for these. In the meantime in order to execute and consume CDP commands I am defining the command and response dictionaries manually, and then calling ExecuteCdpCommand on the ChromeDriver as demonstrated above. This has been working well in the interim and does not need to be awaited.

[eugene159]

I'm getting the same error as topic starter. Tried both LoadingFinished and ResponseReceived events. Still getting InvalidOperationException: A command response was not received: Network.getResponseBody Any idea how to make it work ?

[AndreasKarz]

The only result will be

[AndreasKarz]

Is it really such a rocket science to send back the response body with the status code and all the other information? In Playwright it's one line, but with Selenium it takes a doctor's work.

[titusfortner]

Fwiw, Playwright has Microsoft money and a dozen people assigned to it; all of this .NET code has been written by volunteers.

Where Selenium is supporting the open web and working with browser vendors to try to come up with cross-browser standards for this functionality (which takes a lot of work), the Playwright team is seemingly hostile to that effort.

Which is to say, we would absolutely welcome your assistance to make this functionality better if you would like to make some Pull Requests. Thanks.

[firecrauter]

After a while, I was able to achieve it, at the cost of having to cache the RequestWillBeSentEventArgs, to be able to have it together with the response.

I accept corrections/improvements

        private async Task EnableNetworkDevTool()
        {
            IDevToolsSession session = Driver.GetDevToolsSession();
            DevToolSDomains = session.GetVersionSpecificDomains<DevToolsSessionDomains>();
            await DevToolSDomains.Network.Enable(new EnableCommandSettings() { MaxResourceBufferSize = 100000000 });
            AddHandler_ResponseBody(DevToolSDomains.Network, OnResponseBodyFinished);
        }
        private void AddHandler_ResponseBody(NetworkAdapter network, Action<GetResponseBodyCommandResponse, RequestWillBeSentEventArgs> callback)
        {
            Dictionary<string, RequestWillBeSentEventArgs> all_requests = new();
            List<string> Urls_WhiteList = new() { ".js" };
            network.LoadingFinished += loadingFinishedHandler;
            network.RequestWillBeSent += Network_RequestWillBeSent;
            void Network_RequestWillBeSent(object? sender, RequestWillBeSentEventArgs e)
            {
                if (Urls_WhiteList.Any(e.Request.Url.Contains))
                {
                    all_requests[e.RequestId] = e;
                }
            }
            void loadingFinishedHandler(object? sender, LoadingFinishedEventArgs args)
            {
                Task.Run(async () =>
                {
                    GetResponseBodyCommandResponse responseBody = await network.GetResponseBody(new OpenQA.Selenium.DevTools.V103.Network.GetResponseBodyCommandSettings()
                    {
                        RequestId = args.RequestId
                    });
                    bool exist = all_requests.TryGetValue(args.RequestId, out var requesteventargs);
                    if (exist && requesteventargs != null)
                    {
                        Application.Current.Dispatcher.Invoke(() =>
                        {
                            callback(responseBody, requesteventargs);
                        });
                        all_requests.Remove(args.RequestId);
                    }
                });
            }
        }
        private void OnResponseBodyFinished(GetResponseBodyCommandResponse responseBody, RequestWillBeSentEventArgs requestEventArgs)
        {
            string bodystr = responseBody.Body;
            string url = requestEventArgs.Request.Url;
            //...

        }

[Roman1137]

Still unable to get the response body. The code above does not work for me.

If someone has a working solution, share it please.

[firecrauter]

@Roman1137

Still unable to get the response body. The code above does not work for me.

List<string> Urls_WhiteList = new() { ".js" };

Did you add in the list called "Urls_WhiteList" the url you want?, try add google: List<string> Urls_WhiteList = new() { ".js","google." };

[Roman1137]

@firecrauter, thanks for the reply. I have managed to make it work in this way:

    var enableFetchCommandSettings = new EnableCommandSettings();
    var requestPattern = new RequestPattern();
    requestPattern.RequestStage = RequestStage.Response;
    enableFetchCommandSettings.Patterns = new[] { requestPattern };
    var fetchAdapter = new FetchAdapter(UiTestContext.Current.DevTools.Session);
    fetchAdapter.Enable(enableFetchCommandSettings).GetAwaiter().GetResult();
    
    fetchAdapter.RequestPaused += async (sender, args) =>
    {
        try
        {
            var responseBody = await fetchAdapter.GetResponseBody(new()
            {
                RequestId = args.RequestId
            });

            if (responseBody != null)
            {
                byte[] data = Convert.FromBase64String(responseBody.Body);
                string decodedString = Encoding.UTF8.GetString(data);
               
               // log body
            }
        }
        catch (Exception e)
        {
        }
        finally
        {
            await fetchAdapter.ContinueRequest(new ContinueRequestCommandSettings() { RequestId = args.RequestId });
        }
    };

but, I receive the error when new tab is opened at this row: await fetchAdapter.ContinueRequest(new ContinueRequestCommandSettings() { RequestId = args.RequestId });

The error may be: "Fetch is not enabled", or "InterruptionId is not valid".

[nvborisenko]

It should be resolved by https://github.com/SeleniumHQ/selenium/pull/12701. It's not released yet as far as I understand.

[nvborisenko]

There were a bunch of improvements in DevTools area in the latest version, please take a look. Closing it, feel free to open new issue.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.