Corrupt Chrome profile

[Viliasas]

Hello,

Recently my scripts started to corrupt chrome profile and I'm unable to figure out what's going on. I've tried different undetected-chromedriver and Chrome versions, but the problem persists. I'm not sure if it's really corrupting Chrome profile, but when I start the script with same profile I get Message: unknown error: cannot connect to chrome at 127.0.0.1:50173 errors. And if I delete current Chrome profile and recreate it - it works (for some time). This happens randomly, it might run for 30 items or it might run for 1000.

undetected-chromedriver version: 3.1.5 (tried most of them) Chrome version: 99 (tried to downgrade to 98 and 97) Python version: 3.9.10 OS: Ubuntu Server 20.04 LTS with XCFE4 GUI (machine is accessed through xrdp).

Another thing to note is that I got a lot of items to retrieve and for each of them I need to change proxy which Chrome uses. So after each item retrieval I close Chrome with driver.quit() (as with with it wouldn't close on its own).

Does anyone know what I could be doing wrong or if there are some files in Chrome profile which I could delete to make it work again? To recreate the profile I need to download one extension and log into it, and I haven't automated this part.

This is the full traceback of the exception:

Traceback (most recent call last):
  File "/home/rpa/Desktop/automation/rpa-products/main.py", line 86, in <module>
    collector.collect()
  File "/home/rpa/Desktop/automation/rpa-products/services/data_collector.py", line 63, in collect
    self.collect_seller_data(lead, True)
  File "/home/rpa/Desktop/automation/rpa-products/services/data_collector.py", line 74, in collect_seller_data
    driver = self.driver.init_chrome_driver(lead['seller_link'])
  File "/home/rpa/Desktop/automation/rpa-products/services/driver.py", line 34, in init_chrome_driver
    return uc.Chrome(options=options, version_main=99)  # , use_subprocess=True
  File "/home/rpa/Desktop/automation/rpa-products/venv/lib/python3.9/site-packages/undetected_chromedriver/__init__.py", line 392, in __init__
    super(Chrome, self).__init__(
  File "/home/rpa/Desktop/automation/rpa-products/venv/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 70, in __init__
    super(WebDriver, self).__init__(DesiredCapabilities.CHROME['browserName'], "goog",
  File "/home/rpa/Desktop/automation/rpa-products/venv/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 93, in __init__
    RemoteWebDriver.__init__(
  File "/home/rpa/Desktop/automation/rpa-products/venv/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 269, in __init__
    self.start_session(capabilities, browser_profile)
  File "/home/rpa/Desktop/automation/rpa-products/venv/lib/python3.9/site-packages/undetected_chromedriver/__init__.py", line 579, in start_session
    super(selenium.webdriver.chrome.webdriver.WebDriver, self).start_session(
  File "/home/rpa/Desktop/automation/rpa-products/venv/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 360, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/home/rpa/Desktop/automation/rpa-products/venv/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 425, in execute
    self.error_handler.check_response(response)
  File "/home/rpa/Desktop/automation/rpa-products/venv/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:52429
from chrome not reachable
Stacktrace:
#0 0x55caa8bfc7d3 <unknown>
#1 0x55caa895851f <unknown>
#2 0x55caa8946879 <unknown>
#3 0x55caa897ef49 <unknown>
#4 0x55caa8976e66 <unknown>
#5 0x55caa89b2355 <unknown>
#6 0x55caa89ac5a3 <unknown>
#7 0x55caa8981ddc <unknown>
#8 0x55caa8982de5 <unknown>
#9 0x55caa8c2d49d <unknown>
#10 0x55caa8c4660c <unknown>
#11 0x55caa8c2f205 <unknown>
#12 0x55caa8c46ee5 <unknown>
#13 0x55caa8c23070 <unknown>
#14 0x55caa8c62488 <unknown>
#15 0x55caa8c6260c <unknown>
#16 0x55caa8c7bc6d <unknown>
#17 0x7f26050ed609 <unknown>

patcher__del__ called

[17ss]

i have same issue, i deleted profile directory and rebuild, it will be work.

[Viliasas]

It seems that deleting Default/Preferences fixes the profile, but then it forgets about installed extension.

A workaround for my case is to delete Default/Preferences and then to load extension with --load-extension argument. The only downside of that is that I need to deal with annoying extension installed welcome screen. But I can switch to original tab... And the downside of that is that it uses additional proxy bandwidth to load that welcome page.

[sebdelsol]

If you have your own fork you can merge #558 that should fix the corrupted Default/Preferences

[Viliasas]

Thanks for the help @sebdelsol . I don't have a fork, but I've implemented a workaround in my script which will help it to run while your fix gets to the main repository. At this point I've prepared a Chrome profile and I backed up its Default/Preferences file. Once I encounter WebDriverException, I delete current Preferences file and copy over previously prepared one in its place.

[17ss]

Since commented lines from 366 to 380 in undetected_chromedriver/__init__.py, the issue has not recurred.

[sebdelsol]

@17ss : the code you've commented removes the restore pop up. If you want to keep this code, then add fs.truncate() just after json.dump(config, fs) that should do the trick.

[17ss]

@sebdelsol: First of all thank you very much, I tried the patch(#558) you provided, but it didn't work for me, the issue still occurs. so i comment those codes.

[sebdelsol]

I can't reproduce your issue even with an extension loaded...

This error is thrown when you try to open Chrome with a profile already in use... have you checked you're not spawning several Chrome all using the same profile directory (or the one in use by your regular Chrome browser currently running) ?

[Viliasas]

@sebdelsol In my case I'm opening and closing Chrome often. Maybe at some point when I try to open it again Chrome is not fully closed or maybe it needs more time to clean up after its previous closing.

[Viliasas]

No, I don't use prefs. All modifications are done with add_argument and execute_cdp_cmd. But thanks, I will try to delete that folder once the script crashes and I will check if it works.

On Thu, Mar 17, 2022 at 5:50 PM sebdelsol @.***> wrote:

Are you using options.add_experimental_options("prefs", ...) ? it breaks the Default/Preferences file when it already exists (it should merge nested dictionaries which is not the case at the moment). So a workaround would be to delete the Default/Preferences in your profile.

— Reply to this email directly, view it on GitHub https://github.com/ultrafunkamsterdam/undetected-chromedriver/issues/554#issuecomment-1071003966, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATI67D6NDCJDUONIW4TXI3VANIGDANCNFSM5Q7BJA2Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you authored the thread.Message ID: @.*** com>

[lihanchen]

I can confirm that the patch doesn't work for the latest UC version (3.5.5). I have to comment out the try-catch block that opens Default/Preferences file as @17ss do. The code to comment out is here: https://github.com/ultrafunkamsterdam/undetected-chromedriver/blob/0aa5fbe252370b4cb2b95526add445392cad27ba/undetected_chromedriver/init.py#L427

[asapsmc]

I'm also getting my Chrome profiles corrupted in macOS, ie, I loose access to Chrome Profiles at once, and I have to start from scratch configuring them again in Google Chrome. I'm not sure exactly at what point in code this happens, but this happened 2 times in the same afternoon, so I quitted trying again. At some point, I received some messages selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:52429 from chrome not reachable, but I'm not sure if I always receive that message before/during corruption. Here is my code:

import json
import os
import subprocess
import tempfile
from functools import reduce

import undetected_chromedriver as uc
from selenium import webdriver


class ChromeWithPrefs2(uc.Chrome):
    # FIXME: This is to solve problem with setting Chrome experimental options
    # This workaround is from https://github.com/ultrafunkamsterdam/undetected-chromedriver/issues/524
    def __init__(self, *args, options=None, **kwargs):
        if options:
            self._handle_prefs(options)

        super().__init__(*args, options=options, **kwargs)
        self.keep_user_data_dir = False

    @staticmethod
    def _handle_prefs(options):
        if prefs := options.experimental_options.get("prefs"):
            # turn a (dotted key, value) into a proper nested dict
            def undot_key(key, value):
                if "." in key:
                    key, rest = key.split(".", 1)
                    value = undot_key(rest, value)
                return {key: value}

            # undot prefs dict keys
            undot_prefs = reduce(
                lambda d1, d2: {**d1, **d2},  # merge dicts
                (undot_key(key, value) for key, value in prefs.items()),
            )

            user_data_dir = None
            for arg in options.arguments:
                if arg.startswith('--user-data-dir='):
                    user_data_dir = arg.split('=', 1)[1]
                    break

            # If no user_data_dir was provided, create a temporary one
            if not user_data_dir:
                user_data_dir = os.path.normpath(tempfile.mkdtemp())
                options.add_argument(f"--user-data-dir={user_data_dir}")

            # create the preferences json file in its default directory
            default_dir = os.path.join(user_data_dir, "Default")
            if not os.path.exists(default_dir):
                os.makedirs(default_dir)

            prefs_file = os.path.join(default_dir, "Preferences")
            with open(prefs_file, encoding="latin1", mode="w") as f:
                json.dump(undot_prefs, f)

            del options._experimental_options["prefs"]


applescript = '''
        tell application Google Chrome
            quit
        end tell
    '''
subprocess.run(['osascript', '-e', applescript])

options = webdriver.ChromeOptions()
options.headless = False
options.add_argument("--log-level=DEBUG")
options.add_argument("--user-data-dir=/Users/machine/Library/Application Support/Google/Chrome")
options.add_argument(f'--profile-directory={CHROME_PROFILES[profile]}')
prefs = {
    "download.default_directory": '/Users/machine/Downloads',
    "download.prompt_for_download": False,
    "download.directory_upgrade": True,
    "plugins.always_open_pdf_externally": True,
    "safebrowsing.enabled": True,
}
options.add_argument('--handle_prefs')
options.add_experimental_option("prefs", prefs)
options.add_argument("--kiosk-printing")

driver = ChromeWithPrefs2(options=options, use_subprocess=True)