Scrapy+LUA - Error happened while executing Lua script : cannot select the specified elements

[maciejbak85]

I am scraping LinkedIn profile, I would like to click a button ( Show More ), but then I get errors..

in general scrapper works fine, logs in, scrape the data, scrolling the page works fine too ( to load all the data ). Only issue I have is with this button to show more. He is my script:

function main(splash, args)
    splash:init_cookies(splash.args.cookies)

    url=args.url
    headers = {
        ["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36"
    }
    splash:set_custom_headers(headers)

    splash.private_mode_enabled = false
    assert(splash:go(url))

    for i = 1, 10 do
        splash.scroll_position = {y=i*500}
        splash:wait(1)
    end

    --splash:runjs("document.querySelector('.pv-experience-section__see-more.pv-profile-section__actions-inline.ember-view button').click();")

    local buttons = splash:select_all('.pv-experience-section__see-more.pv-profile-section__actions-inline.ember-view button')
    for i, button in inpairs(buttons) do
        button:mouse_click()
    end
    splash:wait(2)

    local entries = splash:history()
    local last_response = entries[#entries].response
    return {
      url = splash:url(),
      headers = last_response.headers,
      http_status = last_response.status,
      cookies = splash:get_cookies(),
      html = splash:html(),
      png = splash:png()
    }
end

as you can see, Ive tried clicking button also with splash:runjs ( which works fine while checking in browser console ). Runjs does not shows me any errors, but well just does not work. selecting buttons and trying to click them shows me an lua script error:

2020-07-23 09:12:00 [scrapy_splash.middleware] WARNING: Bad request to Splash: {'error': 400, 'type': 'ScriptError', 'description': 'Error happened while executing Lua script', 'info': {'message': '[string "function main(splash, args)..."]:20: cannot select the specified elements', 'type': 'SPLASH_LUA_ERROR', 'splash_method': 'select_all', 'source': '[string "function main(splash, args)..."]', 'line_number': 20, 'error': 'cannot select the specified elements'}}

I am sure it is there as I was checking screenshots. Anyone can help me how to fix this ? thanks!