requests-html icon indicating copy to clipboard operation
requests-html copied to clipboard

Cromium sessions are not closed

Open ElLorans opened this issue 5 years ago • 6 comments

After using r.html.render() Cromium sessions are not closed

ElLorans avatar Aug 07 '19 12:08 ElLorans

I've noticed similar issues in a project I'm currently working on (Python 3.6.5, Mac OS X) Ultimately, the issue hangs chromium, blocking the rest of the application, or ultimately leads to the following error: pyppeteer.errors.BrowserError: Browser closed unexpectedly: [0830/120316.855779:ERROR:gpu_process_transport_factory.cc(1007)] Lost UI shared context. [warn] kqueue: Too many open files in system [err] evsignal_init: socketpair: Too many open files in system I don't yet know what to do to resolve, but more than happy to provide test cases, logs, or assist in troubleshooting.

rjjanuary avatar Aug 30 '19 17:08 rjjanuary

@ElLorans @rjjanuary To close all windows you need to call .session.close()

webpage = HTMLSession().get(url)
webpage.html.render()
# Do what you need
webpage.session.close()

I tried also .close(), .html.browser.close(), .html.browser.disconnect() but they don't close everything.

NeuroFox avatar Sep 08 '19 03:09 NeuroFox

If you're reusing the same HTMLSession to render multiple HTML pages, consider using context manager syntax instead. In this context, closing the session too early, will cause later requests to fail down the line, and the application to possibly deadlock if you're using a thread pool.

Consider the following example, where calling .close() instead of .session.close() is the correct behavior:

with HTMLSession() as session:
    webpage1 = session.get(url1)
    webpage1.html.render()
    webpage1.close()
    webpage2 = session.get(url2)
    webpage2.html.render()
    webpage2.close()
    return blahblahblah

tianqiyuan avatar Jun 18 '20 09:06 tianqiyuan

I tried the same but ram keeps geeting fuller and fuller until no ram is available. My code:

# async function
async def get_address ():

    b = await asession.get(url)
    await b.html.arender(sleep=1,keep_page=True)
    b.close()

# create list with n functions 
list = []
for i in range(10):
    list.append(get_address)
i=0
# run script
while(1):
    i+=1
    logger.info('start')
    asession.run(*list)
    logger.info('finished '+str(i)+' th time')

update: removed keep_page - now session get closed ;b

a14stoner avatar Jul 19 '21 05:07 a14stoner

perfect solution https://blog.csdn.net/wgPython/article/details/102519342

alimehran200 avatar Jan 23 '23 09:01 alimehran200

I ran into this same issue just today! Full code is a bit more complex, but essentially this worked:

r = HTMLSession().get
#stuff I do with the session in here
r.session.close()

geofflambeth avatar Jan 31 '24 04:01 geofflambeth