selenium latePageScript Javascript

[angelo337]

Hi there I am trying to crawl a Site with Norconex V3, this site:

use a lot of JS to make interaction and content, I am trying to crawl content however I am not able to figure out the click on late script, could you please point me an documento or an example?, of ways to interact with the Selenium driver. thanks a lot angelo

[essiembre]

The WebDriverHttpFetcher does not really offer ways to directly interact with Selenium. Instead, it lets you write JavaScript that can simulate user behaviour. In your case, you seem to want to simulate a button click. It could look somewhat like that.

window.onload = function() {
      let button = document.getElementById(id);
      button.click();
};

It could get tricky to get the timing right (when your script is triggered vs when the DOM will be read by the crawler). Since <latePageScript> will be added after a page has been loaded, it is possible it launches after the crawler has read the loaded page DOM already. If this happens, you may want to use <earlyPageScript> instead. Given the example script I shared, it will be added before the page is done loading, but executed when all elements are loaded.

Depending on your page gets rendered, you may have to play with the various timeout options, to make sure there is enough time to perform your operations.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.