Clarify what happens when additional driver commands are executed

[antonky]

Let's say I got back my response, now I want to use the driver to do some additional things like clicking buttons or scrolling etc.

    def get_table(self, response):
        driver = response.request.meta["driver"]
        iframe = driver.find_element_by_id("iframed-stats")
        driver.switch_to.frame(iframe)
        division = Select(driver.find_element_by_id("ddlDiv"))
        division.select_by_value("MBN")
        driver.implicitly_wait(10)
        division = Select(driver.find_element_by_id("ddlCat"))
        division.select_by_value("A1")
        driver.implicitly_wait(10)

Now, I want to select something from the document as it is after all of this has been executed. It doesn't seem like request gets updated as I run these commands, so am I correct in my understanding that I would now have to do this

s = scrapy.Selector(text=driver.page_source)
s.css(".class")

[shawnCaza]

This is a good question. I've been puzzling over this today.

@REMitchell has a scrapy lesson where she seems to suggest you can just switch back to use response with scrapy after using driver commands with selenium.

However, I've not been able to make it work in my application. Response was just giving me an a bunch of white spaces with the method in the screenshot. Your solutions is giving me the results I expect.

Thank you.