Listen to network requests and get their Body "Fetch domain is not enabled"

[MisterSzled]

I'm currently trying to listen to all the requests made by a page.

I've managed to get to a point where I can list all the requests but I'm failing to get their body.

My results typically look like:

Response received for request RequestId("9233.182"): https://en.wikipedia.org/static/images/project-logos/enwiki.png
Failed to get response body: Error -32000: Fetch domain is not enabled

This error: Failed to get response body: Error -32000: Fetch domain is not enabled is the problem as I need to be able to scrape some JSON and I'm not sure how to correct it. I'm guessing it's an easy fix with some CDP command but I can't see it. Any help would be really appreciated.

I've created a minimal example of what i've done so far here:

use chromiumoxide::browser::{Browser, BrowserConfig};
use chromiumoxide::handler::viewport::Viewport;
use futures::StreamExt;

use chromiumoxide::cdp::browser_protocol::network::EventResponseReceived;
use chromiumoxide::cdp::browser_protocol::fetch::GetResponseBodyParams;

#[async_std::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
        let mut viewport = Viewport::default();
        viewport.width = 1920;
        viewport.height = 1200;
        let (mut browser, mut handler) = Browser::launch(
                BrowserConfig::builder()
                        .with_head()
                        .viewport(viewport)
                        .window_size(1920, 1200)
                        .build()?,
        )
        .await?;

        let handler_task = async_std::task::spawn(async move {
                while let Some(event) = handler.next().await {
                        if let Err(e) = event {
                                eprintln!("Handler error: {}", e);
                                break;
                        }
                }
        });

        let page = browser.new_page("https://en.wikipedia.org").await?;

        let mut network_events = page.event_listener::<EventResponseReceived>().await?;
        let network_task = async_std::task::spawn({
                let page = page.clone();
                async move {
                        while let Some(event) = network_events.next().await {
                                println!(
                                        "Response received for request {:?}: {}",
                                        event.request_id, event.response.url
                                );

                                let body = page
                                        .execute(GetResponseBodyParams::new(event.request_id.clone()))
                                        .await;
                                match body {
                                        Ok(response_body) => {
                                                println!(
                                                        "Response body (base64 encoded = {}): {}",
                                                        response_body.base64_encoded,
                                                        response_body.body
                                                );
                                        }
                                        Err(e) => eprintln!("Failed to get response body: {}", e),
                                }
                        }
                        Ok::<(), Box<dyn std::error::Error + Send + Sync>>(())
                }
        });

        page.find_element("input#searchInput")
                .await?
                .click()
                .await?
                .type_str("Rust programming language")
                .await?
                .press_key("Enter")
                .await?;

        // Wait for the navigation and then get the page content.
        let html = page.wait_for_navigation().await?.content().await?;
        println!("Page content length: {}", html.len());

        browser.close().await?;

        // Await the spawned tasks.
        network_task.await;
        handler_task.await;

        Ok(())
}

[MisterSzled]

I've been tracking through the docs and have enabled .enable_request_intercept() in my browser builder now.

However when trying to listen to the events mentioned as:

let mut browser_auth_requests = browser.event_listener::<EventAuthRequired>().await?;
let auth_tasks = async_std::task::spawn({
        async move {
                while let Some(event) = browser_auth_requests.next().await {
                        println!(
                                "Response received for request {:?}",
                                event.request_id
                        );
                }
                Ok::<(), Box<dyn std::error::Error + Send + Sync>>(())
        }
});

I don't see any events.

I have tried attaching the listener to the page instead of the browser but I can't define a new page without a valid url and since I have .enable_request_intercept() enabled every request requires auth which I can't provide since the event listener isn't yet attached...

I tried a similar set of code from: #254 but I think it's the same problem I'm having here.

[linkinlin]

chromiumoxide::cdp::browser_protocol::network::GetResponseBodyParams √ use chromiumoxide::cdp::browser_protocol::fetch::GetResponseBodyParams ×