Ziinc

is there any specific reason why chrome/puppeteer instead of selenium?

While there is no doubt that puppeteer would be great for browser automation and there isn't much browser monoculture issues (since they are adding in firefox soon), I am not...

we could consider microsoft's playwright, which is cross browser. Might make rotating the user agent easier

making hash functions optional is a possibility, for users that would like to trade correctness for memory improvements - [ ] add in hash function to middleware - [ ]...

@jack-s9 thanks for the fix, code indentation does not actually require backticks, see here https://hexdocs.pm/elixir/1.13/writing-documentation.html could you remove the backticks and add in the additional line after the header? i...

@oltarasenko coveralls uploading doesn't work for PRs, for some reason.

To clarify, when I say handle, I mean "to do something with the request, like update a database)

Hmmm. This would mean a middleware piping step prior to piping the success response through the item pipeline. Let's call this hypothetical step Response Middlewares. Should the sole responsibility of...

There are two ways: 1. adding them in `start_urls` 2. Incrementally adding them in `parse_item` which method depends on how many url permutations you are looking at. if the number...

I don't think it'll be good to replicate efforts with a separate ui solution, as much of the functionality is already in the crawly_ui repo. I think slowly breaking up...