crawl4ai
crawl4ai copied to clipboard
[DOUBT] Pagination Supported ?
It would be great if it supports pagination . I wanted to use this solution to scrape whole lengthy documentations of some great open source projects for making easy knowledge base which i can feed to llm for getting answer's related to that project or using it as updated knowledgebase for llm (as you know llm info can get outdated very quickly) . This will become great knowledge base creation kind of solution for me personally plus many others can use for same task as knowledge is and will be the limiting factor with llm's altho llm's can be connected with internet with tools still giving full knowledge base is much better because llm might not take wise decision in getting correct required info from internet plus there are many other reasons too :-))