Ziinc

@jfmlima hi this is a bug, thanks for the report. It is caused by this line here: https://github.com/oltarasenko/crawly/blob/8c8b3651559529bcb81ec1477ade18386f794f14/lib/crawly/request.ex#L57 @oltarasenko the requests creation might need some refactoring. As there is no...

I see, yes, setting naming is an issue, but that's what the docs are for :sweat_smile: Interesting that the spider doesn't auto close when there isn't any requests in the...

@tanguilp could you clarify what you mean by a cdp driver?

I think let's narrow down this issue to **the ability to track url redirections through a form of url history**. Currently hackney does not store intermediate locations (https://github.com/benoitc/hackney#automatically-follow-a-redirection), so a...

#170 fixes point 1 and 2. Only making of file logger backend an opt in dep remains.

Crawly automatically starts a cowboy http endpoint to allow remote controlling of the Crawly service. It is probably a good idea to let this as an opt-in feature, to avoid...

https://hexdocs.pm/crawly/basic_concepts.html#behaviour-functions

- [ ] add in examples - [ ] add to https://hexdocs.pm/crawly/basic_concepts.html#behaviour-functions, link to callback docs - [ ] add to https://hexdocs.pm/crawly/Crawly.Spider.html#c:init/1 , main doc reference.

https://stackoverflow.com/questions/30267943/elixir-download-a-file-image-from-a-url Use a custom pipeline to manage the downloading . In your spider, scrape the media urls and pass it as a nested map key. Then pattern match on it....

@oltarasenko sounds like a good idea, i'll think a bit more about the api and update here. I should have time for it in the coming weeks. @s0kil i think...