linkedin-jobs-scraper icon indicating copy to clipboard operation
linkedin-jobs-scraper copied to clipboard

Timeout on loading job details

Open stoodkev opened this issue 3 years ago • 4 comments

I've tried the Usage example code provided, with and without the cookie and always end up getting timeouts, any idea?

scraper:info Env variable LI_AT_COOKIE detected. Using LoggedInRunStrategy +0ms
  scraper:info Setting chrome launch options {
  headless: true,
  args: [
    '--enable-automation',
    '--start-maximized',
    '--window-size=1472,828',
    '--lang=en-GB',
    '--no-sandbox',
    '--disable-setuid-sandbox',
    '--disable-dev-shm-usage',
    '--disable-gpu',
    '--disable-accelerated-2d-canvas',
    '--disable-setuid-sandbox',
    '--disable-dev-shm-usage',
    "--proxy-server='direct://",
    '--proxy-bypass-list=*',
    '--allow-running-insecure-content',
    '--disable-web-security',
    '--disable-client-side-phishing-detection',
    '--disable-notifications',
    '--mute-audio',
    '--lang=en-GB'
  ],
  defaultViewport: null,
  pipe: true,
  slowMo: 100
} +2ms
Listening on port 3001
  scraper:info [Engineer][Europe] Starting new query: query="Engineer" location="Europe" +1s
  scraper:info [Engineer][Europe] Query options {
  locations: [ 'Europe', 'United States' ],
  limit: 33,
  optimize: true,
  filters: { type: [ 'F', 'C' ] }
} +0ms
  scraper:info Setting authentication cookie +2s
  scraper:info [Engineer][Europe] Opening https://www.linkedin.com/jobs/search?keywords=Engineer&location=Europe&f_JT=F%2CC&start=0 +204ms
  scraper:info [Engineer][Europe] Session is valid +3s
  scraper:info [Engineer][Europe] Jobs fetched: 7 +308ms
  scraper:error [Engineer][Europe][1] Timeout on loading job details +0ms
  scraper:error [Engineer][Europe][1] Timeout on loading job details +4s
  scraper:error [Engineer][Europe][1] Timeout on loading job details +4s
  scraper:error [Engineer][Europe][1] Timeout on loading job details +4s
  scraper:error [Engineer][Europe][1] Timeout on loading job details +4s
  scraper:error [Engineer][Europe][1] Timeout on loading job details +4s
  scraper:error [Engineer][Europe][1] Timeout on loading job details +4s
  scraper:info [Engineer][Europe][1] Pagination requested (2) +30s

stoodkev avatar May 18 '21 04:05 stoodkev

Try to use a greater slowMo value:

const scraper = new LinkedinScraper({
        headless: true,
        slowMo: 250,
});

spinlud avatar May 22 '21 08:05 spinlud

I have been facing the same issue. Tried it with greater slowMo values but it hasn't helped. When run as an anonymous session, it shows "No Jobs found" and with an authenticated session, shows "Timeout on loading job details"

maharshi66 avatar Jun 07 '21 07:06 maharshi66

I've fixed the issue for a while using Tor as proxy. Now I ve got the issue again, even by setting a high slowMo (2000), I fetch a few jobs and then it stops working with Too many requests error.

stoodkev avatar Jul 01 '21 09:07 stoodkev

Linkedin is becoming very aggressive with rate-limiting on anonymous sessions. I suggest to use an authenticated session if you can, limits are much less strict

spinlud avatar Jul 01 '21 09:07 spinlud