how to get response payload ?

[rOY369]

Trying this for monitoring requests

@browser()
def scrape_responses_task(driver: Driver, data):
    # Define a handler function that will be called after a response is received
    def after_response_handler(
        request_id: str,
        response: cdp.network.Response,
        event: cdp.network.ResponseReceived,
    ):
        # Extract URL, status, and headers from the response
        url = response.url
        status = response.status
        headers = response.headers
        
        # Print the response details 
        print(
            "after_response_handler",
            {
                "request_id": request_id,
                "url": url,
                "status": status,
                "headers": headers,
            },
        )

        # Append the request ID to the driver's responses list
        driver.responses.append(request_id)

    # Register the after_response_handler to be called after each response is received
    driver.after_response_received(after_response_handler)

    # Navigate to the specified URL
    driver.get("https://example.com/")

    # Collect all the responses that were appended during the navigation
    collected_responses = driver.responses.collect()
    
    # Save it in output/scrape_responses_task.json
    return collected_responses

Looked at whats available in the response object, the response payload/body/text is not available ?

would really appreciate any help how to get the response data ?

[rOY369]

my bad. the final collected_responses itself contains everything.

[SpaceX-RM]

I am facing the same problem

[rOY369]

@SpaceX-RM i was able to get the responses with the same above snippet included in the documentation. what problem are you facing?

[SpaceX-RM]

@rOY369 I get this error:

Traceback (most recent call last): File "C:\Users\hp\AppData\Local\Programs\Python\Python310\lib\site-packages\botasaurus\browser_decorator.py", line 201, in run_task result = func(driver, data) File "c:\Users\hp\Desktop\Scripts\hdfc copy 3.py", line 37, in scrape_responses_task collected_responses = driver.responses.collect() File "C:\Users\hp\AppData\Local\Programs\Python\Python310\lib\site-packages\botasaurus_driver\driver.py", line 2050, in collect return self.driver.collect_responses(self) File "C:\Users\hp\AppData\Local\Programs\Python\Python310\lib\site-packages\botasaurus_driver\driver.py", line 853, in collect_responses return [self.collect_response(request_id) for request_id in request_ids] File "C:\Users\hp\AppData\Local\Programs\Python\Python310\lib\site-packages\botasaurus_driver\driver.py", line 853, in return [self.collect_response(request_id) for request_id in request_ids] File "C:\Users\hp\AppData\Local\Programs\Python\Python310\lib\site-packages\botasaurus_driver\driver.py", line 850, in collect_response return response UnboundLocalError: local variable 'response' referenced before assignment Task failed for input: None View screenshot at C:\Users\hp\Desktop\Scripts\error_logs\2025-09-08_16-46-29\screenshot.png We've paused the browser to help you debug. Press 'Enter' to close.