R Max Espinoza comments

Results 56 comments of


                                            R Max Espinoza

Add Kafka Topic Integration to Scrapy Redis

I'd suggest to create it as standalone library.

[Question] Fetch request url from redis fail

Thank you for your patience. V0.8.0 has been released 🎉

I'm not familiar with playwright but [scrapy-selenium](https://github.com/clemfromspace/scrapy-selenium/blob/develop/scrapy_selenium/middlewares.py#L13) could by a good of example of how to integrate headless browsers.

make_request_from_data implementation in RedisMixin

At the time, `request_from_dict` required several keys as you can see [here](https://github.com/scrapy/scrapy/commit/cec36a9284641bb7b69a2081ad92cd7d4ee25934#diff-e06f4f2d459a0d000958dc12cdc96e94d127adc947617c9f39c540a3c7536ee3L49). And the need was to load start URLs from redis, which initially were just URL strings, and then...

[dev] Optimize batch fetch method to boost throughput

> @rmax How do you think about this implementation, it disabled `spider_idel` usage. I wonder if we need a switch between `spider_idle` and `fill_requests_queue`. Interesting the use of the other...

Provide better error handling

Example error: https://github.com/rolando/scrapydo/issues/7

Added: Made it able to suppress the warnings when a "normal" request is yielded

Thanks. Looks good to me. Could you add a simple test for the no-warnings flag?

Add a note about using `twisted.internet.defer.inlineCallbacks` directly instead of `inline_requests`

@HelloEdit If I remember correctly, if you use `@inlineCallbacks` then you have to return deferreds and to return the final item you would use `defer.returnValue(item)`.

Include AUTHORS and COPYING files in the source distribution

Friendly nudge.

R Max Espinoza

Upgrading jsonschema to 4.x