in_threads icon indicating copy to clipboard operation
in_threads copied to clipboard

A way to wrap the default thread executor

Open ximus opened this issue 2 years ago • 2 comments

a ruthlessly handy library thank you,

I make use of it to speed up generating database seeds in a rails app (db/seeds.rb) file.

as such:

users = [
  {
    display_name: 'John McTest',
    email: '[email protected]',
    password: password,
  },
  # ...
].in_threads.with_progress('users').map do |entry|
  User.create!(entry)
end

In order to make use of database connections across threads, one needs to wrap each block with ActiveRecord::Base.connection_pool.with_connection.

For readability concerns, I would like to avoid sprinkling ActiveRecord::Base.connection_pool.with_connection throughout my seeds file.

Right now I use the following heavy handed global workaround:

require 'in_threads'

InThreads::Pool.class_eval do
  def run(&block)
    wrapped = ->{ ActiveRecord::Base.connection_pool.with_connection { block.call } }
    @queue.push(wrapped)
  end
end

I was wondering if there is a better, recommended way of handling this situation? Maybe a way to configure (or wrap) the default thread execution? Or if I could easily write a new in_db_threads helper that would behave like in_threads, but wrap with activerecord, but that seems impossible without monkey patching this gem.

ximus avatar Jul 23 '22 05:07 ximus

Thank you 😄

Currently library doesn't have any supported way to hook into the process. The workaround looks fine, only problem it being a monkey patch if/when something changes. I checked introducing another wrapper (.in_threads.with_progress('users').with_db.), but looks like it will not be simple, so best way would be to introduce hooks like .in_threads(around_each: proc{ |&block| ….with_connection(&block) }) and then you could add a method like in_db_threads to wrap it.

toy avatar Jul 23 '22 15:07 toy

Thanks for the response.

If comes a stage when I need to get rid of the patch, I'll happily try implementing that hook idea. I'll let you decide if you want to keep this issue open to track the idea.

ximus avatar Jul 23 '22 17:07 ximus