skrape.it icon indicating copy to clipboard operation
skrape.it copied to clipboard

[QUESTION] Are there cases where BrowserFetcher does not fully support CSR?

Open pistolcaffe opened this issue 1 year ago • 1 comments

describe what you want to archive I am going to create a user guide page for my app and I need to crawl that page in my app. (I need to crawl certain urls in the app as well as notion pages) https://fundevstudio.notion.site/524eafbfa8f2414898d6d8d79f222c05?pvs=4

However, even if i use the initial BrowserFetcher,cannot get the title of the loaded page.

Please let me know if there is any additional way I can do it.

Code Sample

fun main(args: Array<String>) {
    skrape(BrowserFetcher) {
        request {
            url = "https://fundevstudio.notion.site/524eafbfa8f2414898d6d8d79f222c05?pvs=4"
        }

        response {
            htmlDocument {
                println("title: $titleText")
            }
        }
    }
}

[expect] title: 인사이트 플로우 가이드 [but] title: Notion – The all-in-one workspace for your notes, tasks, wikis, and databases.

If it is not possible, waitUntill property value similar to playwright, puppeteer: load, networkidle, documentLoaded Please consider providing options.

pistolcaffe avatar Jun 24 '23 06:06 pistolcaffe