openaustralia
Sidekiq processes using 100% CPU
This is causing the queue to build up because the workers take forever to finish running scrapers. Right now there's 63 scrapers queued but only 2 containers actually running.
Extracted from #992.
I found some duplicate jobs on the queue. After clearing them out there's a lot more running containers.
So the duplicates definitely make the sidekiq process run glacially but I've also found a bunch of new scrapers that are in infinite loops and logging output for every loop too. That's bound to peg the CPU but what should we do in that case? I think the only solution would be to add some kind of CPU time restriction. It would be a shame to add another constraint.