undetected-chromedriver
undetected-chromedriver copied to clipboard
Webpage doesn't fully load using the get() function
I'm trying to scrap some html elements from a webpage, but the problem is that when I use the get() funciton to load the webpage then find the html elements, the execution of the get() function finishes before the page being fully load. Thus, the html elements I'm searching for ain't being found.
Any idea why this problem is happening? And is there any solution other than waiting for a fixed amount of time after the get() function?
Minimum reproducable code:
import undetected_chromedriver as uc
import time
from selenium.webdriver.common.by import By
def main():
driver = uc.Chrome()
t1 = time.time()
driver.get(
'https://kazan.domclick.ru/search?deal_type=sale&category=living&offer_type=flat&sale_price__gte=1&sale_price__lte=1000000000&offset=0')
print('loaded in', time.time() - t1, 'seconds')
print('found elements:', len(driver.find_elements(By.XPATH, '//div[@data-test="offers-list__item"]')))
if __name__ == '__main__':
main()
The webpage in this case loads in one second and finds 0 elements
An unefficient solution:
import undetected_chromedriver as uc
import time
from selenium.webdriver.common.by import By
def main():
driver = uc.Chrome()
t1 = time.time()
driver.get(
'https://kazan.domclick.ru/search?deal_type=sale&category=living&offer_type=flat&sale_price__gte=1&sale_price__lte=1000000000&offset=0')
time.sleep(10)
print('loaded in', time.time() - t1, 'seconds')
print('found elements:', len(driver.find_elements(By.XPATH, '//div[@data-test="offers-list__item"]')))
if __name__ == '__main__':
main()
Now it finds 10 elements because the webpage was fully loaded.
Is there a way to fix this issue without having to wait a fixed time?
using wait
Please check how to use explicit and implicit waits in Selenium.