Is it possible to run Chrome browser in headless mode?

[saisyam]

I am trying to run the Chrome browser in headless mode, by adding the following options:

options = Options() options.add_argument("--headless")

But the application starts and stops immediately with an exception Element not found. Is there a plan to implement this in near future?

[thalissonvs]

Hi @saisyam , the headless mode should be working. Can you share your code? Also, maybe you're getting blocked. Set a custom useragent when using headless

[saisyam]

import asyncio
from pydoll.browser.chrome import Chrome
from pydoll.constants import By
from pydoll.browser.options import Options

async def scrape_articles():
    options = Options()
    options.add_argument("--headless")
    async with Chrome(options) as browser:
        await browser.start()
        page = await browser.get_page()
        await page.go_to("https://example.com")  # Replace with actual website URL
   
asyncio.run(scrape_articles())

Here is my code..

[saisyam]

Got it, I figured it out to run in headless mode. But getting the following error:

Exception ignored in: <finalize object at 0x104a6d020; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.10.16/Frameworks/Python.framework/Versions/3.10/lib/python3.10/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/Cellar/[email protected]/3.10.16/Frameworks/Python.framework/Versions/3.10/lib/python3.10/tempfile.py", line 868, in _cleanup
    cls._rmtree(name, ignore_errors=ignore_errors)
  File "/opt/homebrew/Cellar/[email protected]/3.10.16/Frameworks/Python.framework/Versions/3.10/lib/python3.10/tempfile.py", line 864, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "/opt/homebrew/Cellar/[email protected]/3.10.16/Frameworks/Python.framework/Versions/3.10/lib/python3.10/shutil.py", line 725, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/opt/homebrew/Cellar/[email protected]/3.10.16/Frameworks/Python.framework/Versions/3.10/lib/python3.10/shutil.py", line 664, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/opt/homebrew/Cellar/[email protected]/3.10.16/Frameworks/Python.framework/Versions/3.10/lib/python3.10/shutil.py", line 662, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 66] Directory not empty: 'Default'

Is it because of not closing the browser object properly? I tries await page.close() and await browser.stop() but no clean exit.

[thalissonvs]

Yes, this is a bug on the context manager,. I'll soon open a PR to close this problem. You can try to handle this in a more 'manual' way, something like:

browser = Chrome(options)
await browser.start()
...
await browser.stop()
temp_dirs = browser._temp_directory_manager.temp_dirs # this is the list of temp dirs. You can clean it using another approach

If that's work for you, please, close the issue as resolved :) You can open another issue reporting specifically this bug

[saisyam]

Great. I am on Mac. Suddenly the issue disappeared. I will raise a bug if I see it again. Closing it now.