Paweł Miech
Paweł Miech
In scrapyd/eggstorage.py we use LooseVersion from distutils https://github.com/scrapy/scrapyd/blob/b45c20547b91f34938ee574f86ba966d4b17b05c/scrapyd/eggstorage.py#L37 The distutils package is deprecated and slated for removal in Python 3.12. [PEP 632](https://www.python.org/dev/peps/pep-0632/) recommends using [packaging.version](https://packaging.pypa.io/en/latest/version.html). There is one problem, that...
Before switching to github actions, we were not running tests on Windows as far as I know. After switching to github actions most tests were passing on Windows, but coverage...
fixes #117 This fixes important security problem https://github.com/scrapy/scrapyd/issues/117 and also adds unit tests for webservice, increases coverage to 80%. There is some discussion there in issue about allowing some characters,...
I have simple Twisted resource looking like this ``` python import sys from twisted.web import server from twisted.web.resource import Resource from twisted.internet import reactor from twisted.python import log from twisted.internet...
Currently offsite middleware reads allowed domains from spider attribute on spider opened and uses that to decide whether request should be followed or not. https://github.com/scrapy/scrapy/blob/129421c7e31b89b9b0f9c5f7d8ae59e47df36091/scrapy/spidermiddlewares/offsite.py#L58 I have use case where...
Are there any specific features or bug fixes that you (users, developers, anyone) consider important, top priority? Please share information here in thread. If someone would like to volunteer to...
Since version 3.0 there are restrictions if Request from generator has callback/errback. Why is it like this? What is the reason for this change? I have some spiders like this...
I have a project with tests inheriting from both twisted.trial.unittest.TestCase and tests not inheriting from anything, just plain Python objects. When running without asyncio reactor, they all work fine. When...
This is because of https://github.com/TeamHG-Memex/sklearn-crfsuite/issues/68
Library not compatible with Python 3.10, that's because of this problem [in python-crfsuite](https://github.com/scrapinghub/python-crfsuite/issues/130)