scrapy-inline-requests icon indicating copy to clipboard operation
scrapy-inline-requests copied to clipboard

Add a note about using `twisted.internet.defer.inlineCallbacks` directly instead of `inline_requests`

Open rmax opened this issue 9 years ago • 3 comments

rmax avatar Oct 18 '16 03:10 rmax

Could you provide an example please? I try to use @inlineCallbacks without success.

poupryc avatar Jan 01 '20 14:01 poupryc

@HelloEdit If I remember correctly, if you use @inlineCallbacks then you have to return deferreds and to return the final item you would use defer.returnValue(item).

rmax avatar Jan 02 '20 21:01 rmax

I think I've managed otherwise, without inline_request.

# -*- coding: utf-8 -*-
import scrapy
import re
from twisted.internet.defer import inlineCallbacks

from sherlock import utils, items, regex


class PagesSpider(scrapy.spiders.SitemapSpider):
    name = 'pages'
    allowed_domains = ['thing.com']
    sitemap_follow = [r'sitemap_page']

    def __init__(self, site=None, *args, **kwargs):
        super(PagesSpider, self).__init__(*args, **kwargs)

    @inlineCallbacks
    def parse(self, response):
        # things
        request = scrapy.Request("https://google.com")
        response = yield self.crawler.engine.download(request, self) 
        # Twisted execute the request and resume the generator here with the response
        print(response.text)

poupryc avatar Jan 02 '20 21:01 poupryc