requests-html
requests-html copied to clipboard
Cromium sessions are not closed
After using r.html.render() Cromium sessions are not closed
I've noticed similar issues in a project I'm currently working on (Python 3.6.5, Mac OS X)
Ultimately, the issue hangs chromium, blocking the rest of the application, or ultimately leads to the following error:
pyppeteer.errors.BrowserError: Browser closed unexpectedly: [0830/120316.855779:ERROR:gpu_process_transport_factory.cc(1007)] Lost UI shared context. [warn] kqueue: Too many open files in system [err] evsignal_init: socketpair: Too many open files in system
I don't yet know what to do to resolve, but more than happy to provide test cases, logs, or assist in troubleshooting.
@ElLorans @rjjanuary To close all windows you need to call .session.close()
webpage = HTMLSession().get(url)
webpage.html.render()
# Do what you need
webpage.session.close()
I tried also .close(), .html.browser.close(), .html.browser.disconnect() but they don't close everything.
If you're reusing the same HTMLSession to render multiple HTML pages, consider using context manager syntax instead. In this context, closing the session too early, will cause later requests to fail down the line, and the application to possibly deadlock if you're using a thread pool.
Consider the following example, where calling .close() instead of .session.close() is the correct behavior:
with HTMLSession() as session:
webpage1 = session.get(url1)
webpage1.html.render()
webpage1.close()
webpage2 = session.get(url2)
webpage2.html.render()
webpage2.close()
return blahblahblah
I tried the same but ram keeps geeting fuller and fuller until no ram is available. My code:
# async function
async def get_address ():
b = await asession.get(url)
await b.html.arender(sleep=1,keep_page=True)
b.close()
# create list with n functions
list = []
for i in range(10):
list.append(get_address)
i=0
# run script
while(1):
i+=1
logger.info('start')
asession.run(*list)
logger.info('finished '+str(i)+' th time')
update: removed keep_page - now session get closed ;b
perfect solution https://blog.csdn.net/wgPython/article/details/102519342
I ran into this same issue just today! Full code is a bit more complex, but essentially this worked:
r = HTMLSession().get
#stuff I do with the session in here
r.session.close()