Increased memory use when using `bin/jobs` to start Solid Queue?

[jp524]

I'm working on a project that was using Solid Queue v0.7.0 and using bundle exec rake solid_queue:start to start the process. When I updated to v1.0.0 last week, I also changed the command to start Solid Queue to use bin/jobs per current documentation.

I noticed that the memory use of the associated Heroku app was greatly increased. I wasn't sure if this was due to updating the gem or changing the start command, so I tried changing back the command from bin/jobs to the Rake task and have noticed a decrease in memory use, per screenshot below.

I read in the discussion on #343 that using bin/jobs is supposed to help with memory use thanks to eager loading, so I'm a bit puzzled by this situation.

[rosa]

Huh, that's indeed puzzling! May I ask what's your Solid Queue configuration? And whether you have config.rake_eager_load set to true by any chance?

[jp524]

Here's my Solid Queue configuration:

# config/queue.yml
default: &default
  dispatchers:
    - polling_interval: 1
      batch_size: 500
  workers:
    - queues: [ login, background, default ]
      threads: 3
      processes: 1
      polling_interval: 0.1

development:
 <<: *default

test:
 <<: *default

staging:
 <<: *default

production:
 <<: *default

# config/recurring.yml
one:
  class: JobOne
  schedule: every 5 minutes
two:
  class: JobTwo
  schedule: every 75 minutes
solid_queue_cleanup:
  command: "SolidQueue::Job.clear_finished_in_batches"
  schedule: every day at 3 am
  queue: background

I don't have config.rake_eager_load set at all so it defaults to false I believe.

[rosa]

Thanks so much for sending that.

I don't have config.rake_eager_load set at all so it defaults to false I believe.

Yes, that's right. I'm quite puzzled in any case, about this behaviour... But I wonder if, in your case, eager loading is causing that extra memory to be used in the beginning 😕 Would you be up to do a quick test if you can do that without running into trouble with your Heroku instance? That'd be setting

config.rake_eager_load = true

and then checking to see the memory usage invoking Solid Queue with bundle exec rake solid_queue:start.

[jp524]

Here's what I'm seeing after setting config.rake_eager_load = true an hour ago:

It looks like it's using a bit more memory than without eager loading, but it's still not as much as using bin/jobs. (On the graph there's a brief period where we're using bin/jobs again due to an automatic deployment.)

Update, a couple hours later: The memory usage warning are coming back:

[rosa]

Sorry for the delay and thanks for trying that out 🙏 My only guess right now is that it might be because of eager loading the whole app instead of auto-loading. In theory, eager-loading the whole app before forking should help with memory used in total by all processes, but in this case you only have 3 processes and perhaps having them auto-load stuff when they use it instead of eager-loading everything uses less memory. I'm going to do some tests with our app to see if this makes sense, but it's my only guess right now 🤔

[jp524]

Thank you for looking into this @rosa! I'll switch back to using the Rake task to start Solid Queue, but please let me know if you end up finding anything during your testing.