no such session from ChromeDriver (suspect shm size problem of container)

[jasonchanhku]

Hi guys,

Thanks again for adding ChromeDriver. I recently tried running my scraper.py (has selenium) locally on my PC and everything was scraped fine. However, when running it on morph.io, I get the following error:

selenium.common.exceptions.WebDriverException: Message: no such session (Driver info: chromedriver=2.37.544315 (730aa6a5fdba159ac9f4c1e8cbc59bf1b5ce12b7),platform=Linux 4.15.12-x86_64-linode105 x86_64)

Also, Total disk space used was only 58.8 KB

The error would come at inconsistent timings. I suspect it could be something to do with the shm size of docker container, but I could be wrong. An alternative solution would be using Firefox driver. Would that be considered as well since ChromeDriver was recently added ?

Would appreciate it if anyone can shed some light on this. Thanks.

Regards, Jason

[mlandauer]

@jasonchanhku is this the scraper you're talking about? https://morph.io/jasonchanhku/DataScienceDemandScraper

[jasonchanhku]

@mlandauer yes it is. The code line error is inconsistent. Now it says line 60 but sometimes it says line 54. It still works fine when I execute the exact some scraper.py script on my PC. Thanks for the reply.

[jasonchanhku]

for reference to this issue: https://testingrepository.com/fix-no-session-chromedriver-error/

[mlandauer]

@jasonchanhku have you seen https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md#tips? It suggests running chrome with --disable-dev-shm-usage. Could you please try that and let me know if that fixes the problem? If it does I'll update the documentation to help others out.

[jasonchanhku]

@mlandauer thanks for the response. I tried adding the --disable-dev-shm-usage as follows:

options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome(options=options)

However, I am still getting the same error. I am not sure if --disable-dev-shm-usage is among the available list of arguments available based on here. Unless I am not using --disable-dev-shm-usage in the right context.

Do let me know if i am using it wrongly.

[jasonchanhku]

Tried almost everything including the above at this point. Is there anyway I can modify the default shm size of the container ? something tells me perhaps I am not using --disable-dev-shm-usage correctly...

[mlandauer]

Hi @jasonchanhku - sorry to hear about the pain this is causing. I started having another look at this the a few days ago but only had a short amount of time. I think focusing on getting some verbose debugging output out of chromedriver and chrome headless is the way to go. Otherwise we're poking around in the dark a bit.

There's some instructions for doing this in python here: https://sites.google.com/a/chromium.org/chromedriver/logging. What I got stuck on is figuring out how to pipe the logging output to standard out so it's visible in the morph.io scraper output. I tried sending the logging output to a file (as in the example) and then trying to stream the file output to standard out in a seperate thread but I got a bit stuck on my somewhat limited python knowledge. Hopefully you will have more success!