arsenic icon indicating copy to clipboard operation
arsenic copied to clipboard

Published 20 hours ago • verified publisher icon HENNGE

Exception getting new session: io.UnsupportedOperation: fileno

Open Fogapod opened this issue 6 years ago • 11 comments

Versions

OS: Ubuntu 18.04 LTS x86_64 Python: 3.6.5 firefox: 60.0.1 geckodriver 0.20.1 arsenic: 1.0.0.dev8

My code

from arsenic import get_session, services, browsers
service = services.Geckodriver()
browser = browsers.Firefox(firefoxOptions={ 'args': ['-headless']} )


async with get_session(service, browser) as session:
  await session.get('https://google.com')

Traceback

  File "/usr/local/lib/python3.6/dist-packages/arsenic/__init__.py", line 16, in __aenter__
    self.session = await start_session(self.service, self.browser, self.bind)
  File "/usr/local/lib/python3.6/dist-packages/arsenic/__init__.py", line 28, in start_session
    driver = await service.start()
  File "/usr/local/lib/python3.6/dist-packages/arsenic/services.py", line 99, in start
    self.log_file
  File "/usr/local/lib/python3.6/dist-packages/arsenic/services.py", line 34, in subprocess_based_service
    process = await impl.start_process(cmd, log_file)
  File "/usr/local/lib/python3.6/dist-packages/arsenic/subprocess.py", line 61, in start_process
    stdin=DEVNULL,
  File "/usr/lib/python3.6/asyncio/subprocess.py", line 225, in create_subprocess_exec
    stderr=stderr, **kwds)
  File "uvloop/loop.pyx", line 2368, in __subprocess_run
  File "uvloop/handles/process.pyx", line 566, in uvloop.loop.UVProcessTransport.new
  File "uvloop/handles/process.pyx", line 692, in uvloop.loop.__process_convert_fileno
io.UnsupportedOperation: fileno
  • same exception happens without using uvloop
  • same exception happens using chrome browser

Fogapod avatar May 25 '18 18:05 Fogapod

same exception happens without using uvloop

Could you post the exact error message when not using uvloop? uvloop isn't supported right now because the last time I tried there were some issues that I couldn't work around.

ojii avatar May 28 '18 01:05 ojii

Could you post the exact error message when not using uvloop?

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/arsenic/__init__.py", line 16, in __aenter__
    self.session = await start_session(self.service, self.browser, self.bind)
  File "/usr/local/lib/python3.6/dist-packages/arsenic/__init__.py", line 28, in start_session
    driver = await service.start()
  File "/usr/local/lib/python3.6/dist-packages/arsenic/services.py", line 99, in start
    self.log_file
  File "/usr/local/lib/python3.6/dist-packages/arsenic/services.py", line 34, in subprocess_based_service
    process = await impl.start_process(cmd, log_file)
  File "/usr/local/lib/python3.6/dist-packages/arsenic/subprocess.py", line 61, in start_process
    stdin=DEVNULL,
  File "/usr/lib/python3.6/asyncio/subprocess.py", line 225, in create_subprocess_exec
    stderr=stderr, **kwds)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 1192, in subprocess_exec
    bufsize, **kwargs)
  File "/usr/lib/python3.6/asyncio/unix_events.py", line 200, in _make_subprocess_transport
    **kwargs)
  File "/usr/lib/python3.6/asyncio/base_subprocess.py", line 39, in __init__
    stderr=stderr, bufsize=bufsize, **kwargs)
  File "/usr/lib/python3.6/asyncio/unix_events.py", line 706, in _start
    universal_newlines=False, bufsize=bufsize, **kwargs)
  File "/usr/lib/python3.6/subprocess.py", line 667, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "/usr/lib/python3.6/subprocess.py", line 1184, in _get_handles
    c2pwrite = stdout.fileno()
io.UnsupportedOperation: fileno

Fogapod avatar May 28 '18 05:05 Fogapod

As a workaround, pass a log_file to Geckodriver (or whatever driver you are using). It should be an object that is valid for stdout/stderr as described in https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.AbstractEventLoop.subprocess_exec.

ojii avatar May 28 '18 05:05 ojii

I added log_file parameter, now I have different issue. I get the following output:

2018-05-28 05:42.02 request                        body={"desiredCapabilities": {"browserName": "firefox", "marionette": true, "acceptInsecureCerts": true, "firefoxOptions": {"args": ["-headless"]}}} method=POST url=http://localhost:43633/session
2018-05-28 05:42.02 response                       body={"desiredCapabilities": {"browserName": "firefox", "marionette": true, "acceptInsecureCerts": true, "firefoxOptions": {"args": ["-headless"]}}} data={'value': {'error': 'unknown error', 'message': 'Process unexpectedly closed with status 1', 'stacktrace': ''}} method=POST response=<ClientResponse(http://localhost:43633/session) [500 Internal Server Error]>
<CIMultiDictProxy('Connection': 'close', 'Content-Type': 'application/json; charset=utf-8', 'Cache-Control': 'no-cache', 'Content-Length': '105', 'Date': 'Mon, 28 May 2018 05:42:02 GMT')>
 url=http://localhost:43633/session

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/arsenic/__init__.py", line 16, in __aenter__
    self.session = await start_session(self.service, self.browser, self.bind)
  File "/usr/local/lib/python3.6/dist-packages/arsenic/__init__.py", line 29, in start_session
    return await driver.new_session(browser, bind=bind)
  File "/usr/local/lib/python3.6/dist-packages/arsenic/webdriver.py", line 57, in new_session
    raise SessionStartError(err_resp['error'], err_resp.get('message', ''), original_response)
arsenic.errors.SessionStartError: unknown error: Process unexpectedly closed with status 1

Log file says I don't have display variable defined, but I'm using headless mode

1527486122031   geckodriver     INFO    geckodriver 0.20.1
1527486122039   geckodriver     INFO    Listening on 127.0.0.1:43633                                                    1527486122544   mozrunner::runner       INFO    Running command: "/usr/bin/firefox" "-marionette" "-profile" "/tmp/rust_mozprofile.INA3ouudlnMM"
Error: no DISPLAY environment variable specified

My code:

from arsenic import get_session, services, browsers


service = services.Geckodriver(log_file=open('file.txt','w'))
browser = browsers.Firefox(firefoxOptions={ 'args': ['-headless'] })
async with get_session(service, browser) as session:
  await session.get('https://google.com')

Fogapod avatar May 28 '18 05:05 Fogapod

Looks like the options changed (either in FF or geckodriver). this worked for me:

browser = browsers.Firefox(**{'moz:firefoxOptions': { 'args': ['-headless'] }})

ojii avatar May 28 '18 06:05 ojii

New issue occurred. Log file output:

1527489279485   geckodriver     INFO    geckodriver 0.20.1
1527489279495   geckodriver     INFO    Listening on 127.0.0.1:54365
1527489279997   mozrunner::runner       INFO    Running command: "/usr/bin/firefox" "-marionette" "-headless" "-profile" "/tmp/rust_mozprofile.Tq4p7Bgte7hq"
*** You are running in headless mode.
Assertion failure: [unhandlable oom] Failed to allocate object while tenuring., at /build/firefox-lojCEZ/firefox-60.0.1+build2/js/src/vm/JSContext.cpp:1662

I'm able to get page with synchronous selenium library, so Firefox browser and driver are working

Fogapod avatar May 28 '18 06:05 Fogapod

unhandlable oom

You ran out of RAM? (OOM = Out Of Memory)

ojii avatar May 28 '18 06:05 ojii

Yes, seems like previous firefox sessions remained in memory and ate it. Is this a bug? Output ps -aux | grep fire:

kiwi       385  0.1 10.2 2151272 103512 pts/1  Sl+  06:29   0:04 /usr/lib/firefox/firefox -marionette --headless -profile /tmp/rust_mozprofile.IDL1dqR12NKe
kiwi       452  0.0  2.1 1538776 21312 pts/1   Sl+  06:29   0:00 /usr/lib/firefox/firefox -contentproc -childID 1 -isForBrowser -intPrefs 42:0|44:0|74:0| -boolPrefs 5:1|61:1|276:0|299:0| -stringPrefs 285:36;df6fb09a-00ad-4dfd-a53c-ef0582a18525| -schedulerPrefs 0001,2 -greomni /usr/lib/firefox/omni.ja -appomni /usr/lib/firefox/browser/omni.ja -appdir /usr/lib/firefox/browser 385 true tab
kiwi       505  0.0  1.6 1524400 16328 pts/1   Sl+  06:29   0:00 /usr/lib/firefox/firefox -contentproc -childID 2 -isForBrowser -intPrefs 42:0|44:0|74:0| -boolPrefs 5:1|61:1|276:0|299:0| -stringPrefs 285:36;df6fb09a-00ad-4dfd-a53c-ef0582a18525| -schedulerPrefs 0001,2 -greomni /usr/lib/firefox/omni.ja -appomni /usr/lib/firefox/browser/omni.ja -appdir /usr/lib/firefox/browser 385 true tab
kiwi       538  0.2 10.8 2179688 109572 pts/1  Sl+  06:30   0:05 /usr/lib/firefox/firefox -marionette --headless -profile /tmp/rust_mozprofile.7monsLLmwYcX
kiwi       588  0.0  9.2 1813584 93352 pts/1   Sl+  06:30   0:01 /usr/lib/firefox/firefox -contentproc -childID 1 -isForBrowser -intPrefs 42:0|44:0|74:0| -boolPrefs 5:1|61:1|276:0|299:0| -stringPrefs 285:36;2fd196cf-10f5-46d7-b129-6416e05f7d59| -schedulerPrefs 0001,2 -greomni /usr/lib/firefox/omni.ja -appomni /usr/lib/firefox/browser/omni.ja -appdir /usr/lib/firefox/browser 538 true tab
kiwi       627  0.0  1.6 1524400 16348 pts/1   Sl+  06:30   0:00 /usr/lib/firefox/firefox -contentproc -childID 2 -isForBrowser -intPrefs 42:0|44:0|74:0| -boolPrefs 5:1|61:1|276:0|299:0| -stringPrefs 285:36;2fd196cf-10f5-46d7-b129-6416e05f7d59| -schedulerPrefs 0001,2 -greomni /usr/lib/firefox/omni.ja -appomni /usr/lib/firefox/browser/omni.ja -appdir /usr/lib/firefox/browser 538 true tab
kiwi      1569  0.0  0.1  14856  1120 pts/3    S+   07:14   0:00 grep --color=auto fire

After killing these I was able to navigate and get screenshot. Thanks for the support

Fogapod avatar May 28 '18 07:05 Fogapod

Yes, seems like previous firefox sessions remained in memory and ate it. Is this a bug?

I don't know. I ran the code and it worked fine. Maybe it failed to clean up in the cases where it triggered one of the bugs/problems you ran into? I had to use docker since I don't have a native linux handy, so it cleaned up after each run.

ojii avatar May 28 '18 07:05 ojii

I managed to run session using uvloop setting log_file argument to os.devnull

Fogapod avatar May 28 '18 11:05 Fogapod

I managed to run session using uvloop setting log_file argument to os.devnull

Awesome! I'll try to add it as a fully supported loop then (with CI etc)

ojii avatar May 28 '18 14:05 ojii