Scweet
Scweet copied to clipboard
Update utils.py
I made some changes to the utils folder to avoid some frequently occurring errors. I have to mention that I only made the changes to get_users_follow since I was only using this part. But solutions can also be added to other parts For one to avoid the error in #105 and #122 StaleElementReferenceException I looped through the parts where the error occurred with the help of selenium.common.exceptions - StaleElementReferenceException "try: .... except StaleElementReferenceException:". Since the error popped up at different points of the code I added the loops at several parts. On the other hand to avoid data loss, when an Exception occurs I added the lines
except Exception as e:
print(f"An exception occurred while trying to scroll the page: {e}")
continue
follows_users[user] = follows_elem
except Exception as e:
print(f"An exception occurred while crawling {user}: {e}")
# Call the function inside a try-except block
try:
results = get_users_follow(users, headless, env, follow, verbose, wait, limit)
except Exception as e:
print(f"An exception occurred: {e}")
since I also had issues with the headless mode I added some other options:
options.add_argument('--disable-gpu')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--no-sandbox')
Here I am not sure if it is necessary but it solved my problems.
Just a suggestion and hope It helps others to run the code more smoothly.