Mikhail Korobov
Mikhail Korobov
@foromer4 Scrapy has two request queues - in Scheduler and in Downloader. Downloader uses a small in-memory buffer for requests (its size is equal to CONCURRENT_REQUESTS); it is the component...
The approach looks good 👍 It seems the typing becomes more tricky now, especially as we have py.typed in scrapy now - it affects the user code, not just the...
> What do you think should be tested, at least as a first step? I think we can do it gradually - start with the fixes, as compared to the...
> Should the typing tests run on 3.8 or 3.12? We need different syntax for them. Not sure. Is it because the error messages differ, or is it about something...
@wRAR > We can, though we run the test on a single Python version anyway, so we know which message will be produced. This works. Do you think thought that...
It seems it'd be nice to include the spider name to all log messages, optionally. Probably, it's a matter of allowing to include spider_name variable in the LOG_FORMAT option (https://docs.scrapy.org/en/latest/topics/settings.html#log-format),...
Thanks @smartass101 for providing a workaround! Why not have relevant CSS fixed by default? Are there cases when it is desirable to truncate the information in table, be it a...
> If all time values we use and compare have precision only in seconds, time ticking for less that a second will not cause problems. Wouldn't it be an issue...
What's the workaround for the users if the issue happens (e.g. because extraction takes more than a second)? Is it to change either frozen_time or dateDownloaded manually to make them...
Partially addressed by https://github.com/scrapinghub/scrapy-poet/pull/122.