gpt-crawler icon indicating copy to clipboard operation
gpt-crawler copied to clipboard

Published 20 hours ago • verified publisher icon BuilderIO

maxpage did not work well

Open bcharityfi opened this issue 1 year ago • 1 comments

I set maxpage as 100, 300, or 1000, but it always stopped at 56.

was there anyone to encounter the same issue with me?

And any solution you may have?

bcharityfi avatar Dec 12 '23 06:12 bcharityfi

Hello, I encountered the same situation, but I don't think this is a bug. It might be a feature that needs to be added in the future.

This is because the gpt-crawler can only crawl static pages, and it cannot automatically crawl pages that require interaction with dynamic content. Therefore, it won't reach the set maxpage.

PipeDream941 avatar Dec 22 '23 03:12 PipeDream941