Tangui
Tangui
etags between a compressed and an uncompressed response with same response content should be different per https://datatracker.ietf.org/doc/html/rfc7232#section-2.3.3 As far as I understand cowboy's compress handler is not compliant with this...
When compression is enabled, cowboy only sets `accept-encoding` in the list of headers of the `vary` header when content is actually compressed. That is, a response to a request without...
Hi, Here https://github.com/ExHammer/hammer/blob/master/lib/hammer/backend/ets.ex#L138 we can see that the ETS table is an `ordered_set`. It seems to me that ordered_set are slower than `set` in most cases, except when doing traversal...
Hi, One improvement idea: allowing any key type (any erlang term) for keys, and letting the backend to deal with it if unsupported. For instance, if I want to throttle...
Using `ENV` is unnecessary and causes a bug when deploying with `scrapyd-deploy` because of https://github.com/scrapy/scrapy/blob/master/scrapy/utils/project.py#L80-L98
Usually a fragment leads to the same page.
When a request is being redirected and redirection is handled by the fetcher directly, it is necessary to add both the initial and the landing URL to the list of...
As far as I understand, requests are necessarily stored in a GenServer's state (`Crawly.RequestStorageWorker`) and it is not possible to plug its own storage module. This has some disadvantages: -...
Requests are filtered before being added to the request storage, so as to discard irrelevant pages. When crawling large sites, some filtering rules may be added after crawling is started....
Instead of storing the whole URL in the spider's state in `Crawly.Middlewares.UniqueRequest`, I suggest hashing the URL and storing it instead. I guess taking the first quarter of a sha256...