Celery task : after driver.get() the driver.requests value is None

[enflo]

After performing a driver.get() to a URL, through a Celery task the value of driver.requests is always a empty list. If I make the same call from python driver.requests it does have the values.

Any ideas?

SELENIUM CONFIG FILE

from seleniumwire import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.chrome.options import Options

from config.settings import chromedriver_path


capabilities = DesiredCapabilities.CHROME
capabilities["goog:loggingPrefs"] = {"performance": "ALL"}

opts = Options()
opts.add_argument("--window-size=1920,1055")
opts.add_argument("--incognito")
#opts.add_argument("--headless")
opts.add_experimental_option("excludeSwitches", ["enable-automation"])
opts.add_experimental_option('useAutomationExtension', False)
opts.add_argument("--disable-blink-features=AutomationControlled")
opts.add_argument("--disable-web-security")
opts.add_argument('--no-sandbox')
opts.add_argument('--ignore-certificate-errors-spki-list')
opts.add_argument('--ignore-ssl-errors')


SELENIUM_CHROME_DRIVER = webdriver.Chrome(
    executable_path=chromedriver_path,
    desired_capabilities=capabilities,
    options=opts,
)

Code example:

     SELENIUM_CHROME_DRIVER.get(self.url)

       if check_exists_by_xpath(SELENIUM_CHROME_DRIVER, "//button[contains(.,'Aceptar y cerrar')]"):
           close_cookie_modal = WebDriverWait(SELENIUM_CHROME_DRIVER, DEFAULT_SELENIUM_DELAY).until(
               EC.presence_of_element_located((By.XPATH, "//button[contains(.,'Aceptar y cerrar')]")))
           close_cookie_modal.click()
           print("Close Cookie Modal")

       time.sleep(5)
       print("Requests ???")
       print(SELENIUM_CHROME_DRIVER)
       print(SELENIUM_CHROME_DRIVER.requests)
       for request in SELENIUM_CHROME_DRIVER.requests:
           print(request)
           if request.response:
               print(
                   request.url,
                   request.response.status_code,
                   request.response.headers['Content-Type'],
                   request.response.headers['set-cookie']
               )

[wkeeling]

Thanks for raising this.

Is your Celery task just importing SELENIUM_CHROME_DRIVER from the config file? Does it make any difference if you move the creation of the driver to inside the task - just above where the code uses it? I'm just trying to understand whether having the driver external to the task (assuming it is) is causing the problem.

[rlimaeco]

The same is happening here, and put the creation of the driver inside the task, but still only empty list in driver.requests

[enflo]

I have the driver generated in a file and I import it. I also tried moving the driver code into the Celery task, but the error is the same.

[alicanyuksel]

It's normal. You should define selenium wire machine address in your config. What does it mean ? It means: where your code is executed exactly

So it's potentially your celery container.

Ex:

seleniumwire_options_dict = {
        "auto_config": False,
        "addr": [SELENIUM_WIRE_MACHINE_IP],
    }

If your containers are connected in swarm, you can use directly your celery container name as hostname or localhost if you do some test locally.

driver = webdriver.Chrome(
       **,
        seleniumwire_options=seleniumwire_options_dict,
    )