Running on Server

[AkshayAdiga1997]

Really appreciate your work. I have a question. This works taking cookies. So how do you run this on Ubuntu server?

[austinoboyle]

I haven't run this on a server before, but you will likely have to install something like Xvfb. See the discussion here, and let me know if you get this working! I have a couple of scraping projects that I would love to run on dedicated servers.

[austinoboyle]

Okay, can confirm this will work on a server. This is what setup looked like for me on Centos7:

$ wget https://chromedriver.storage.googleapis.com/2.38/chromedriver_linux64.zip
$ unzip chromedriver_linux64.zip
$ sudo mv ./chromedriver /usr/local/bin
$ wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
$ sudo yum localinstall ./google-chrome-stable_current_x86_64.rpm
$ sudo yum install Xvfb
$ Xvfb :99 -ac &
$ export DISPLAY=:99

Then I pip installed and ran the scrape_linkedin package as per normal

[AkshayAdiga1997]

Thank you!

[dsc03]

@austinoboyle to clarify, for the list of commands you sent above, these are commands that I would need to ssh into my remote server and run/configure there correct? also, 'DISPLAY' looks like an environment variable correct?

[austinoboyle]

@dsc03 Yes that's correct. This was for a Centos server, but should work similarly on Ubuntu/etc with the distro's respective package manager to get xvfb

[dsc03]

Hey @austinoboyle ,

So I've been trying to get this thing setup on my linux server, and when I run sudo yum localinstall ./google-chrome-stable_current_x86_64.rpm, I get the following issues:

Error: Package: google-chrome-stable-69.0.3497.81-1.x86_64 (/google-chrome-stable_current_x86_64)
           Requires: libappindicator3.so.1()(64bit)
Error: Package: google-chrome-stable-69.0.3497.81-1.x86_64 (/google-chrome-stable_current_x86_64)
           Requires: libnss3.so(NSS_3.22)(64bit)
Error: Package: google-chrome-stable-69.0.3497.81-1.x86_64 (/google-chrome-stable_current_x86_64)
           Requires: libXss.so.1()(64bit)
Error: Package: google-chrome-stable-69.0.3497.81-1.x86_64 (/google-chrome-stable_current_x86_64)
           Requires: libgdk-3.so.0()(64bit)
Error: Package: google-chrome-stable-69.0.3497.81-1.x86_64 (/google-chrome-stable_current_x86_64)
           Requires: xdg-utils
Error: Package: google-chrome-stable-69.0.3497.81-1.x86_64 (/google-chrome-stable_current_x86_64)
           Requires: libatk-1.0.so.0()(64bit)
Error: Package: google-chrome-stable-69.0.3497.81-1.x86_64 (/google-chrome-stable_current_x86_64)
           Requires: libgdk_pixbuf-2.0.so.0()(64bit)
Error: Package: google-chrome-stable-69.0.3497.81-1.x86_64 (/google-chrome-stable_current_x86_64)
           Requires: libgtk-3.so.0()(64bit)
Error: Package: google-chrome-stable-69.0.3497.81-1.x86_64 (/google-chrome-stable_current_x86_64)
           Requires: libssl3.so(NSS_3.28)(64bit)
Error: Package: google-chrome-stable-69.0.3497.81-1.x86_64 (/google-chrome-stable_current_x86_64)
           Requires: libatk-bridge-2.0.so.0()(64bit)
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

It seems like I can skip around, which I tried and after following the rest of the commands and trying to execute my code, I was running into the following issue when I tried running my function that uses your package:

[Errno 2] No such file or directory: '/home/wsgi/.google-cookie'

Any thoughts? I've been at it for a bit now so hoping you have some insight. Thanks again.

-Daniel

[austinoboyle]

You likely need to go through each of those dependencies and install it using yum. You may need to enable certain yum repositories to install them. See this link as an example for libappindicator https://stackoverflow.com/questions/49965949/where-is-libappindicator3-so-1