worker
worker copied to clipboard
MontFerret
Chrome no headless option
it might be nice to have the possibility for the build to choose a non-headless for some usecases
Non-headless is not supported by Docker. If you want to use Worker with open Chrome, you need to run the browser locally with open debugging port and let the Worker to connect to it.
Headfull
Hey ! Sadly not for the moment, I'm stuck on this :
- https://github.com/siomiz/chrome/issues/26
- https://forums.docker.com/t/docker-chrome-headfull-with-xvfb-and-remote-debugging-active-dbus/108202
Everything start but chrome api couldn't be called 🤔
I've stoped my research on this last answer :
- https://stackoverflow.com/questions/49285658/how-to-solve-docker-issue-failed-to-connect-to-bus-no-such-file-or-directory
Maybe i'm missing something 🤷♂️
If you want to test i probably have my test branch somewhere based on https://github.com/siomiz/chrome , ping me on discord
Headless
For these reasons I began to study concealing a headless chrome based on https://github.com/paulirish/headless-cat-n-mouse, with an MITM docker before page explorer interpretation. This allow us JS injection to bypass client side headless detection. The cat n mouse race to hide headless is not ended, but we should share it. I've added you to my private repo example, we could integrate it as an option in worker or side by side (DockerCompose) or in ferret (like a cdp option).
Actually my example success on this test : https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html
More informations : https://intoli.com/blog/not-possible-to-block-chrome-headless/
(I've added an fql example on discord to test chrome based on intoli test)
Take a look at: https://github.com/jvitor83/chrome-debug-node-vnc
It is a built-in chrome with debug capabilities and VNC for remote viewer and node to use puppeteer. In some testing it isn't detected as a bot because it is headfull (but inside docker container).
@PierreBrisorgueil hey, have you managed to create a Docker image with non-headless Chromium?
@ziflex Nop, I set up a MITM to hide the headless with multiple JS tricks, but I still haven't managed to make the headfull in docker.
I also wanted to watch how they handled the head full on the crawlee side
Take a look at: https://github.com/jvitor83/chrome-debug-node-vnc
It is a built-in chrome with debug capabilities and VNC for remote viewer and node to use puppeteer. In some testing it isn't detected as a bot because it is headfull (but inside docker container).
works, thx
