Scweet icon indicating copy to clipboard operation
Scweet copied to clipboard

Update utils.py

Open Zabina opened this issue 2 years ago • 0 comments

I made some changes to the utils folder to avoid some frequently occurring errors. I have to mention that I only made the changes to get_users_follow since I was only using this part. But solutions can also be added to other parts For one to avoid the error in #105 and #122 StaleElementReferenceException I looped through the parts where the error occurred with the help of selenium.common.exceptions - StaleElementReferenceException "try: .... except StaleElementReferenceException:". Since the error popped up at different points of the code I added the loops at several parts. On the other hand to avoid data loss, when an Exception occurs I added the lines

                    except Exception as e:
                        print(f"An exception occurred while trying to scroll the page: {e}")
                        continue

            follows_users[user] = follows_elem
        except Exception as e:
            print(f"An exception occurred while crawling {user}: {e}")

    # Call the function inside a try-except block
    try:
        results = get_users_follow(users, headless, env, follow, verbose, wait, limit)
    except Exception as e:
        print(f"An exception occurred: {e}")

since I also had issues with the headless mode I added some other options:

        options.add_argument('--disable-gpu')
        options.add_argument('--disable-dev-shm-usage') 
        options.add_argument('--no-sandbox')

Here I am not sure if it is necessary but it solved my problems.

Just a suggestion and hope It helps others to run the code more smoothly.

Zabina avatar Feb 17 '23 16:02 Zabina