Fred Wu
Hmmm, this is fairly specific, when I have time I'd like to make the tokenizer configurable so that you can supply your own. :)
Hi, if you could issue a PR that would be awesome! 👍
Hi @happysalada, thanks for doing more investigation! To be honest I haven't had chance to use my library for a while so I don't remember much off the top of...
Hmm interesting, nice find! I do have one question though, what happens if a worker dies? Would the entire process stop?
Thanks @rhnonose! I'll need to fix up the test suite (broken due to newer version of Elixir, unrelated to this PR) and after that I'll merge this one in.
The test suite is now fixed on master. Any more info on @robvolk's issue on this branch?
> I'm not sure if I need to reference @rhnonose's `OPQ` directly anymore, but I did need it back when he created his branch for `crawler`. This is no longer...
@adamelmore Hmm, I don't believe there's been any progress on this...
Heya, if I understand your requirement correctly, you don't need to "track" the URLs. The filter is called on every URL. You pass in a module because the filter needs...
Hey @robvolk, how are you using Crawler, as in, with what options? I recommend you use the `:interval` option to rate limit, if you haven't already. As for the second...