undetected-chromedriver icon indicating copy to clipboard operation
undetected-chromedriver copied to clipboard

Webpage doesn't fully load using the get() function

Open repeating opened this issue 2 years ago • 2 comments

I'm trying to scrap some html elements from a webpage, but the problem is that when I use the get() funciton to load the webpage then find the html elements, the execution of the get() function finishes before the page being fully load. Thus, the html elements I'm searching for ain't being found.

Any idea why this problem is happening? And is there any solution other than waiting for a fixed amount of time after the get() function?

Minimum reproducable code:

import undetected_chromedriver as uc
import time
from selenium.webdriver.common.by import By

def main():
    driver = uc.Chrome()
    t1 = time.time()
    driver.get(
        'https://kazan.domclick.ru/search?deal_type=sale&category=living&offer_type=flat&sale_price__gte=1&sale_price__lte=1000000000&offset=0')
    print('loaded in', time.time() - t1, 'seconds')
    print('found elements:', len(driver.find_elements(By.XPATH, '//div[@data-test="offers-list__item"]')))

if __name__ == '__main__':
    main()

The webpage in this case loads in one second and finds 0 elements

An unefficient solution:

import undetected_chromedriver as uc
import time
from selenium.webdriver.common.by import By

def main():
    driver = uc.Chrome()
    t1 = time.time()
    driver.get(
        'https://kazan.domclick.ru/search?deal_type=sale&category=living&offer_type=flat&sale_price__gte=1&sale_price__lte=1000000000&offset=0')
    time.sleep(10)
    print('loaded in', time.time() - t1, 'seconds')
    print('found elements:', len(driver.find_elements(By.XPATH, '//div[@data-test="offers-list__item"]')))

if __name__ == '__main__':
    main()

Now it finds 10 elements because the webpage was fully loaded.

Is there a way to fix this issue without having to wait a fixed time?

repeating avatar Aug 07 '22 17:08 repeating

using wait

yeucodonvn avatar Aug 08 '22 03:08 yeucodonvn

Please check how to use explicit and implicit waits in Selenium.

sebdelsol avatar Aug 19 '22 11:08 sebdelsol