Crawl button with javascript navigation

[hamzamac]

Hi, we are try to crawl a site that use s with Javascript based navigation instead of links of tags. The JavaScripts code controlling the behavior of the buttons is hosted on a different domain (CDN) from that of the target site. When replaying the WACZ, everywhere we have buttons instead of links result into en error, it can not reach the link to JavaScript file.

How can we crawl such a website with Browsertrix-crawler?

[tw4l]

Hi @hamzamac, would you be able to share the URL of the site you're trying to capture so I can take a look?

[hamzamac]

Hi @tw4l, thank you for responding. The site is actually a SharePoint site with MFA. We manages to crawl it by creating a profile. but the links to folders appears to be spans.

when when clicking the button on the replayweb.page it shows this error below (the URL is pointing to is a public CDN URL which is accessible) Do we need to include all the URI for JavaScripts in the seeds?

[tw4l]

Hm, you shouldn't need to include the URIs for scripts - if the script is on the page, the crawler will discover it. This looks to me like it's more likely to be a bug in our replay engine than a missing script. It's hard to tell further without being able to reproduce it ourselves - would you be able to share a copy of the WACZ by email?

[hamzamac]

Hi @tw4l, sure I will send the WACZ to the email on your profile.

[hamzamac]

Hi @tw4l , can you please confirm if you have received the WACZ file? thanks.