Can we make it smart crawler along with sub web crawling ?

[Darshan2104]

Hi, Is it possible to do smart crawl integrating LLM agent.

For example, along with url, use can pass small instruction what actually user wants from webpage.

based on instruction, it will search for those information, if requires access hyperlinks and search for information. and once everything is crawled, it will summarise it based on instruction.