pspsdev

I would be very much interested in that. Hashids is quite a popular library if it would be possible to have efficient cracker it would help a lot of people.

Related: https://github.com/scrapy/scrapyd/issues/197

@jpmckinney just to rule out that cpu is being used by a spider. This can be replicated when scheduling a lot of jobs and polling rate is below a second...

There are also some unmaintained repos that tries to solve this: https://github.com/speakol-ads/scrapyd-redis Simply the sqlite queue is a really bad option for high frequency queues.

I will try to create a demo later, but it's pretty much can be empty scrapyd service running with 1 spider that does nothing. Then creating like 50 schedules per...

Also, in my personal opinion I would say it would make sense to add interface to add your own queue backend instead of doing hacks like those 2 repos mentioned...

And then later sqlite can be switch to some other default is needed, but having a simple method to replace the queue on your own would be a very good...

@jpmckinney thanks, give me a few hours I will try it out.

@jpmckinney I am still doing some tests on my end, give me a few days I will report with more details.