docker-selenium-lambda
docker-selenium-lambda copied to clipboard
umihico
Start xvfb and run chrome without headless
Found your repo really helpful in setting up my selenium pytest tests to run in parallel, thanks! Just thought you might like to know that you can run chrome without the "--headless" if you start Xvfb first. You can then do things like record videos with ffmpeg.
@dbramwell I cannot seem to get XVFB to work with this repo... did you make any changes to your docker file? Feel free to reach out: https://stackoverflow.com/questions/72650323/selenium-xvfb-unable-to-receive-message-from-renderer
Nope, just what's in this PR. Are you getting any specific errors?
Edit (just read the stack overflow):
It could be that you don't have the DISPLAY env var set, so chrome doesn't know what screen to attach to (see here). You'll also need to pass the same display (:99 seems to be convention) to the xvfb wrapper: https://github.com/cgoldberg/xvfbwrapper/blob/master/xvfbwrapper.py#L35
Further edit: It seems like the xvfb wrapper does try to handle setting the DISPLAY env var, so not sure that is the issue
@dbramwell super weird, I even copied the exact code in your PR and still get the "Unable to receive message from renderer" error. Deployed Docker image to Lambda function, maybe that's where the problem is coming from?
Ok, I still don't really understand why it isn't working as is, but:
My automated tests require the browser to be able to play mp4 files. The default chromium builds that this repo uses don' include mp4 codec stuff, so I switched to using the chromium builds from here. This commit makes the code in this PR work successfully, but I am 100% certain that I had xvfb working with the other build of chromium! The only reason I switched was for the MP4 compatibility...
Anyway, it's something for you to go on.
Hello, thank you for your PR and discussions here!
I created another branch and then cherry-picked your first commit. Unfortunately, I got an error on CircleCI. The log in Cloudwatch was below.
IMAGE Launch error: fork/exec /entrypoint.sh: permission denied Entrypoint: [/entrypoint.sh] Cmd: [test.handler] WorkingDir: [/var/task]
I am not sure I will merge this change into main branch, but I am sure that I can keep your successful xvfb version as another branch for other developers.
(I prefer not to use unknown binary if the source code is unclear with less stars. this is why I didn't try your second commit.)
@dbramwell I ran it using the chromium builds that you switched to for mp4 capabilities and it works! Don't know why it was having trouble with the default chromium builds from the repo, but glad the new ones you supplied worked. @umihico this would be a great feature to this repo, seems to be just be a chromium version compatibility issue with XVFB. What version from a trusted source can be used for this?
Good that you have something working @lukehamil55
As I said, I'm so sure I had it working with the regular chromium build. Hopefully I'll get some time to figure it out at some point
@dbramwell appreciate it as always, would this method of spawning a XFVB display directly from a shell file versus in python not work if Selenium creates multiple instances of scrapers using multithreading? If that's the case, then pyvirtualdisplay would be the way to go as you can spawn displays during python runtime especially if multithreading is involved.
Yeah, if you want to start multiple virtual displays you're better off not doing it in the entrypoint. Something like that pyvirtualdisplay library should work fine, not sure how good lambda's multithreading capabilities are though. Is it not simpler to trigger the lambda multiple times and have each invocation just have one scraper instance? Anyway, this is off topic for this repo, pyvirtualdisplay should work fine if you want to use it