Musab Gultekin

I'm thinking to directly sending visited urls without using queue. Why we're using queue? As archivebox already has its own queue.

You can fetch feed using your code, not ```f.ParseURL```. And add this header to your request. ``` req.Header.Set("Accept-Charset", "utf-8") ``` And read response using this auto decoder ``` // Determine...

Yes this could work for any content. But it *removes* not decodes. Anyway, if you are satisfied with this, no problem :)

Because this url returns 403

I don’t know how that will work. Even with CLI interface, some coding needs to be done. For example, extracting data configuration. How can that work, do you have suggestions?...

Its very good feature idea, I'll look if its feasable

Hi William. Thanks for reporting that! Once upon a time, I was working on a simple queue system. Based on chan. Here is the branch: https://github.com/geziyor/geziyor/tree/queue The change is basically...

But max in memory request queue size is hardcoded rn: 1,000,000. We can make it configurable though. Or something dynamic as default, based on the RAM, like Max queue size=...

Yeah, I understand what you mean, So this was not a concern before the queueing system. Since we're planning to integrate that, we should think of ways of doing this...

BUT, if thats the case, why don't the developer does this cancel check just before they queue? And rely on the middleware? Like before even running g.Do method ?