Scrapegraph-ai icon indicating copy to clipboard operation
Scrapegraph-ai copied to clipboard

Published 20 hours ago • verified publisher icon ScrapeGraphAI

Scraping dynamic content

Open JackyZhang1219 opened this issue 1 year ago • 4 comments

Is your feature request related to a problem? Please describe.

Is it possible to get all the dynamically loaded content from a url? E.g. getting all the reviews from IMDB requires clicking "load more" over and over again, and Scrapegraph-ai currently only gets the reviews loaded first

JackyZhang1219 avatar May 21 '24 10:05 JackyZhang1219

not yet

VinciGit00 avatar May 22 '24 09:05 VinciGit00

Hey @JackyZhang1219 thanks we'll look into It! if you have some suggestions write them in the Discord channel

PeriniM avatar May 22 '24 11:05 PeriniM

This is a great feature request. or example, get a 'next page' button to jump to the next list page, like scrapy-slash?

hogking avatar May 28 '24 18:05 hogking

This is a scraper, not a crawler. The way I see it is you can integrate this with scrapy.

angelotc avatar Aug 11 '24 21:08 angelotc

hi please look at the deep scraper graph, is not the solution but it could be a temporal solution

VinciGit00 avatar Oct 19 '24 21:10 VinciGit00