botasaurus icon indicating copy to clipboard operation
botasaurus copied to clipboard

Is it possible to integrate custom scraping task in a class?

Open LeMoussel opened this issue 1 year ago • 0 comments

By integrating custom scraping task _request_scrape_data into a class in this way

class RequestCrawlasaurus():
    def __init__(self, start_url=""):
        self.start_url= start_url

    @request(
            output=None,
            create_error_logs=False,
            raise_exception=True,
            parallel=3,
    )
    def _request_scrape_data(self, request: AntiDetectDriver, link):
        response = request.get(link)

    def crawl(self):
        urls = self._request_scrape_data(link=self.start_url)

I got this error: "RequestCrawlasaurus._request_scrape_data() missing 1 required positional argument: 'link'"

Is it possible to embed the custom scraping task _request scrape data into a class? If yes, how?

LeMoussel avatar Apr 08 '24 09:04 LeMoussel