Concurrency with solid queue not executing the jobs with pause | duration.

[ajays1991]

Hi,

Following is my Job file

# frozen_string_literal: true

class Braze::GasSaver::BrazeUpdateGasSaverJob < ApplicationJob
  queue_as :default

  limits_concurrency to: 1, duration: 4.seconds, key: "braze_gas_saver_update"

  retry_on StandardError, wait: 5.seconds, attempts: 3

  def perform(payload)
    Rails.logger.debug { "Started BrazeUpdateGasSaverJob for #{payload.to_json} at #{Time.zone.today}" }

    web_client = Braze::WebClient.new
    request = Braze::Models::UserRequest.new

    payload.each do |data|
      request.add_attribute(
        external_id: data[:external_id],
        lifetime_sunoco_savings: data[:lifetime_sunoco_savings],
        _update_existing_only: true,
        email: data[:email],
        email_subscribe: data[:email_subscribe],
      )
    end
    begin
      web_client.send(:post, request, Braze::Constants::USER_PATH)
    rescue StandardError => e
      Rails.logger.error("Failed BrazeUpdateGasSaverJob: #{payload.to_json} #{e.message}")
      raise e
    end
  end
end

Following is my solid_queue.yml

default: &default
  dispatchers:
    - polling_interval: 1
      batch_size: 500
      recurring_tasks:
        update_usage_job_p1:
          class: Subscriptions::UpdateAllUsageJob
          schedule: '*/10 0-8 * * *'
        update_usage_job_p2:
          class: Subscriptions::UpdateAllUsageJob
          schedule: '*/10 10-23 * * *'
        nightly_subscription_upgrade_job:
          class: 'Billing::ExecuteNightlyUpgradeJob'
          schedule: '0 8 */1 * *'
        # Below job must run after the above job completes. Which is why they are an hour apart.
        vehicle_subscription_billing_job:
          class: Billing::ExecuteNightlyRunsJob
          schedule: '0 9 */1 * *'
        braze_update_gas_saver_credits:
          class: Braze::GasSaver::BrazeGasSaverBatchJob
          schedule: "30 14 * * *" # This will run everyday at 2:30 PM UTC(9:30 AM EST)

production:
  <<: *default
  workers:
    - queues: [checkout, qualification, '*']
      threads: 5
      processes: 3

staging:
  <<: *default
  workers:
    - queues: [checkout, qualification, '*']
      threads: 5
      processes: 3

development:
  <<: *default
  workers:
    - queues: [checkout, qualification, '*']
      threads: 5
      processes: 3

test:
  <<: *default
  workers:
    - queues: [checkout, qualification, '*']
      threads: 5
      processes: 3

when i equeue jobs like say 100 jobs, i can see the entry in solid_queue_blocked_executions table correct value in concurrency_key. This i debugged on my local. But the jobs are executed in sequence with any pause or delay as set by concurrency settings of 4 seconds.

My question do i need to have a seperate dedicated queue for this concurrency thing or is this fine? If my job file and config file are correct, then please kindly help me rectifying the issue.

Thanks

[rosa]

Hey @ajays1991, first, this configuration is not doing what you think it's doing:

queues: [checkout, qualification, '*']

If the "*" is present, that worker will run all jobs. It doesn't work as first run checkout, then run qualification, then run all jobs. It will run just all jobs, ignoring checkout and qualification. It's equivalent to this:

queues: "*"

That shouldn't affect the concurrency controls, though. I don't quite understand what you mean by

But the jobs are executed in sequence with any pause or delay as set by concurrency settings of 4 seconds.

Duration is not a delay. It's the time that the system guarantees the jobs won't overlap, but if a job finishes in 1 second, the next job will be unblocked and started, as soon as possible. This is all explained here. Does this help?