Sidekiq process got stuck in busy queue

[mohsinssi]

Sidekiq task got stuck sometimes in busy queue when using Async-http and perform multiple simultaneous requests to third party. Some methods i have tried is apply timeout to tasks and also on third party calling.

[ioquatix]

Are you able to give some more details about the nature of the error/hang? Do you have any error logs?

[mohsinssi]

No errors occur however i tried to add custom logs to each function start and end but for some reason second task start log comes but not end log.

Async do
      results = []
      barrier = Async::Barrier.new
      semaphore = Async::Semaphore.new(2, parent: barrier)
      @params.each_slice(500) do |param|
        semaphore.async do
          response = fetch_articles(param)
          body = JSON.parse(response.body)
          results << body
        end
      end
      Rails.logger.debug { 'AsyncHttp#wait' }
      barrier.wait
      process_results(results)
    end

   def fetch_articles(param)
    Rails.logger.debug { "Requesting articles start at #{Time.current}" } // this comes on two times as our params count is more than 500 
    response =  HTTParty call with timeout 6
    Rails.logger.debug { "Requesting articles end at #{Time.current}" } // this comes only one time in logs
    response
  end

[ioquatix]

HTTParty might not be compatible with Async on Ruby 2.x - try using Async::HTTP::Internet.

[mohsinssi]

@ioquatix i have tried that before also the behavior was same.

[ioquatix]

Does it hang every single time or only occasionally?

[mohsinssi]

@ioquatix it hangs occasionally

[ioquatix]

Okay let me see if I can reproduce it.

Can you tell me the exact versions of the gems you are using, e.g. your gem lock file, and specifically sidekiq, sidekiq pro, etc versions, your Ruby version, OS, and async / async-http versions.

[mohsinssi]

Sure sidekiq (6.2.2) sidekiq-cron (1.2.0) async-http (0.56.5) Ruby 3.0.2-alpine3.13 OS Linux

[ioquatix]

Can you tell me Async version too please :)

[ioquatix]

Are you able to try Ruby 3.0.3? There was a few bugs in 3.0.2.

[ioquatix]

(Or try using Ruby 2.7.4 / 2.7.5 which is rock solid).

[mohsinssi]

async (1.30.1) Also not yet tried other ruby versions as i am unable to understand what's causing this issue

[ioquatix]

Ruby 3.0.2 has a bug in IO wait, which can cause intermittent hang, which was fixed in Ruby 3.0.3. I'm not sure if it impacts Async 1.x but it's possible.

[mohsinssi]

ok i will try to upgrade to Ruby 3.0.3 and if issue still comes up i will notify here

[mohsinssi]

@ioquatix I upgraded to Ruby 3.0.3 (also tried 3.1.0) and now the issue changed. After couple of hours sidekiq process get stop and on logs i checked and it shows the same like i mentioned above the request got started but never ended and also cpu spikes up from 15% to 25% suddenly and may be thats the cause for sidekiq process stop.

[mohsinssi]

Here is the latest code

Async do |task|
  task.with_timeout(80) do
    results = []
    barrier = Async::Barrier.new
    semaphore = Async::Semaphore.new(2, parent: barrier)
    internet = Async::HTTP::Internet.new
    @params.each_slice(500) do |param|
      semaphore.async do |sem_task|
        sem_task.with_timeout(10) do
          response = fetch_articles(param, internet)
          body = JSON.parse(response.body)
          results << body
          response&.finish
        end
      end
    end
    Rails.logger.debug { 'AsyncHttp#wait' }
    barrier.wait  (Here it stucked)
    process_results(results)
  ensure
       internet&.close
  end
end

 def fetch_articles(param, internet)
  Rails.logger.debug { "Requesting articles start at #{Time.current}" } // this comes on logs two times as our params count is more than 500
  response =  request using Internet
  Rails.logger.debug { "Requesting articles end at #{Time.current}" }
  response
end

[ioquatix]

Thanks I'll check it.

[mohsinssi]

Thanks I'll check it.

Yes let me know if you found the cause and how i can solve this

[ioquatix]

Can you please try updating to the latest version of async - there was a bug in Async::Barrier that could lead to a hang in some situations.

[ioquatix]

We also found a bug in Ruby's autoload, which can lead to deadlocks, it is fixed in Ruby 3.2. Please open a new issue if you still have issues on the latest release.