Mikhail Korobov

It looks like a good feature to have. Some notes: - Content-Disposition parsing should be moved to a function. - I'm not sure that for missing Content-Disposition filename should be...

@cmeerw what settings would you find reasonable? > CONCURRENT_REQUESTS = 16 Not that it matters much for this discussion, but the CONCURRENT_REQUESTS_PER_DOMAIN is 8 by default, which is similar to...

For the context, on Zyte's hosting solution (Scrapy Cloud) we enable AutoThrottle extension by default; it essentially sets concurrency per domain to 1, and adjusts download delays accordingly.

Hey! Let's keep the conversation calm. Nobody says we don't want to make the change; the change is what's being discussed, right? It is reasonable to have more strict defaults....

I left a couple of miinor comments, but the PR looks good, thanks @whalebot-helmsman !

Hm, that's an interesting approach. Initially I was thinking about keeping the FormRequest class, but replacing the implementation with form2request calls. Is it behavior incompatibility between form2request and current FormRequest...

What's the use case for this? Have you faced any performance issues? I'd be reluctant to accept this change unless there is a real-world benefit. It increases the amount of...

See also: https://github.com/scrapy/scrapy/issues/3867

> Upon DNS errors, a special “unknown_ip_address” is used, instead of the domain name, as suggested in #1659. Should we make this configurable? I think that's fine to keep it...

I like this idea. But we might need to add tests for this proxy implementation (which makes the whole thing larger).