Better support for testing jobs

[nesquena]

From @bradgessler:

A stub could be provided for peeps that want to assert jobs are thrown on the queue in a test env. Make testing a job is performed work easily. Right now I use a hacky thing on my projects:

# Backburner::Worker.enqueue NewsletterSender, [self.id, user.id], :ttr => 1000
Backburner::Worker.class_eval do
  class << self; alias_method :original_enqueue, :enqueue; end
  def self.enqueue(job_class, args=[], opts={})
    job_class.perform(*args)
  end
end

to force the jobs to be executed automatically. Open to the right way to do this that is simple.

[nesquena]

Maybe let people test it with backburner directly. Wiki page on how to stub the job object?

[KrisJordan]

+1 - A wiki page on how you recommend stubbing the job object would be useful. The example above is nice for testing the side-effects of the job, but I could also see it being useful to have an example that confirms a job of a given type was enqueued without actually running it.

[nesquena]

I totally agree, I have it setup in a particular way in most of my applications, I hope to move that onto a wiki page soon.

[ogerman]

Resque has a gem for that purpose: https://github.com/leshill/resque_spec

[nesquena]

Yeah that's the type of thing I'd want to have for backburner as well and I kind of like it as a separate gem.

Nathan Esquenazi

On Thursday, September 19, 2013 at 8:03 AM, ogerman wrote:

Resque has a gem for that purpose: https://github.com/leshill/resque_spec

— Reply to this email directly or view it on GitHub (https://github.com/nesquena/backburner/issues/25#issuecomment-24745718).

[bradgessler]

We use a different backends in quebert, but it adds quite a bit of complexity that's unnecessary for actually running the worker code in production. I really like the idea of a separate gem that's concerned about the complexity of testing.

[kke]

I have config/initializers/backburner.rb that looks like this:

if Rails.env.test?
  module Backburner
    def self.test_mode
      @test_mode ||= :fake
    end

    def self.test_mode=(mode)
      @test_mode = mode
    end

    def self.test_enqueued_jobs
      @test_enqueued_jobs ||= []
    end

    def self.logger
      @logger ||= Logger.new(Rails.root.join('log/backburner_test.log'))
    end

    def self.test_enqueued_jobs_find_by_argument(arg)
      test_enqueued_jobs.find{|j| Array(j[:args]).include?(arg)}
    end

    def self.test_enqueued_jobs_find_by_job(arg)
      test_enqueued_jobs.find{|j| j[:job].eql?(arg)}
    end

    def self.empty_test_queue!
      @test_enqueued_jobs = []
    end

    def self.enqueue(job, *args)
      if args.last.is_a?(Hash)
        options = args.pop
      else
        options = {}
      end
      if args.size.eql?(1)
        args = args.first
      end
      test_enqueued_jobs << {:job => job.to_s, :args => args, :options => options}
      if test_mode.eql?(:fake)
        true
      elsif test_mode.eql?(:inline)
        job.perform(*args)
      else
        raise "Unknown test_mode : #{test_mode}"
      end
    end

    def self.method_missing(meth, *args, &block)
      logger.debug "Backburner method missing: #{meth} : #{args.inspect}"
    end

    class Worker
      def self.enqueue(job, *args)
        if args.last.is_a?(Hash)
          options = args.pop
        else
          options = {}
        end
        if args.size.eql?(1)
          args = args.first
        end
        Backburner.enqueue(job, args, options)
      end

      def self.logger
        Backburner.logger
      end

      def logger
        Backburner.logger
      end

      def self.method_missing(meth, *args, &block)
        logger.debug "Backburner worker method missing: #{meth} : #{args.inspect}"
      end

      def method_missing(meth, *args, &block)
        logger.debug "Backburner worker instance method missing: #{meth} : #{args.inspect}"
      end
    end

    module Queue
      def self.included(base)
        base.extend ClassMethods
      end
      module ClassMethods
        def queue(value=nil)
        end
        def queue_priority(value=nil)
        end
        def queue_respond_timeout(value=nil)
        end
      end
    end
  end
else
  require 'backburner'
  Backburner.configure do |config|
    config.beanstalk_url    = APP_CONFIG["beanstalk"]["servers"]
    config.tube_namespace   = APP_CONFIG["beanstalk"]["namespace"]
    config.on_error         = lambda { |e| DoubleLogger.new(Rails.root.join("log/backburner.log")).error "#{e.message} #{e.backtrace}" }
    config.max_job_retries  = APP_CONFIG["beanstalk"]["max_retries"]
    config.retry_delay      = APP_CONFIG["beanstalk"]["retry_delay"]
    config.default_priority = 1000
    config.respond_timeout  = 3600
    config.default_worker   = Backburner::Workers::ThreadsOnFork
    config.logger           = DoubleLogger.new(Rails.root.join("log/backburner.log"), :log_level => :warn)
    config.primary_queue    = "backburner-jobs"
  end
  module Backburner
    module Queue
      module ClassMethods
        def logger
          Backburner.configuration.logger
        end
      end
    end
    class Job
      def before_perform
        ActiveRecord::Base.verify_active_connections!
      end

      def after_perform
        ActiveRecord::Base.clear_active_connections!
      end
    end
  end
end

This allows me to do something like this in tests:

Backburner.test_mode = :fake
do_something_that_enqueues_something
assert_not_nil Backburner.test_enqeueued_jobs_find_by_job("FooJob")

If you want the jobs to execute inline instantly instead of going to queue you can do

Backburner.test_mode = :inline
do_something_that_enqueues_something
#the job was executed like SomethingJob.perform(args)
#without going to the queue

That code could easily be extracted to a gem you require in test_helper.

[nesquena]

Yep that looks like the kind of easy test helper that should be simple for someone to include. Thanks for posting that!

[nesquena]

Added your suggestion to a gist for safe keeping: https://gist.github.com/nesquena/3b2c445beaa78b53c042

[ogerman]

Once I created a gem, It can be also useful, but it lacks of docs.

[kke]

Excellent

[kuon]

I'm using something a little bit different for testing.

We want to test the "real" thing as much as possible (beanstalkd...).

In test environment I enqueue my jobs as normal, and in my tests, I use:

def background_jobs_wait
  pool = Backburner::Worker.connection
  worker = Backburner.configuration.default_worker.new
  worker.prepare
  pool.tubes.all.each do |tube|
    while tube.peek(:ready)
      worker.work_one_job
    end
  end
end

To process all ready jobs.

I use it like:

it "should send an email" do
  post('/register')
  background_jobs_wait
  # here check that the test email was sent
end

Maybe Backburner could include a way to process all available jobs in a unblocking way?