Best way to play around with Solid Queue in development?

[jeromedalbert]

Solid Queue used to use migrations on a single database, which was nice if you wanted to have everything in the same database in development.

Now Solid Queue uses schemas, which is nice since it does not clutter a brand new Rails 8 app with migrations. The schemas are for separate databases, which makes sense for production (it would be nice to document why by the way, e.g. better isolation or performance). I understand that the library is more intended for production as explained by Rosa:

Solid Queue is more intended for production, Rails ships with another adapter for development, the async adapter.

But it would be nice to be able to easily play around with it in on your local environment, to kick the tires, reproduce production issues in your local, get a feel for how it works with mission_control-jobs, and see if any configuration changes you're making to your Solid Queue setup works correctly on your local before deploying them to minimize issues.

If Solid Queue in development is not a default, then I think it should be easy to set it up on one's local. So far I can think of 3 ways:

1. Keep Solid Queue in a separate schema in production, but one schema in development. I don't know if that's possible. But having everything in one schema keeps the development setup simple, since having multiple databases looks more like a production concern. That would be the simplest setup short of using the async adapter. However it would be harder to remove solid queue (short of using --skip-solid) since I guess you would have to manually edit the schema.

2. Provide an example database.yml setup in some way, like in docs or as comments in that file, so someone can get up and running quickly even if they are not familiar with multi-db setups. This is how I set it up on my local for a new app using postgres:

   default: &default
     adapter: postgresql
     encoding: unicode
     pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
   
   cache: &cache
     <<: *default
     migrations_paths: db/cache_migrate
   queue: &queue
     <<: *default
     migrations_paths: db/queue_migrate
   cable: &cable
     <<: *default
     migrations_paths: db/cable_migrate
   
   development:
     primary:
       <<: *default
       database: myapp_development
     cache:
       <<: *cache
       database: myapp_development_cache
     queue:
       <<: *queue
       database: myapp_development_queue
     cable:
       <<: *cable
       database: myapp_development_cable
   
   test:
     <<: *default
     database: myapp_test
   
   production:
     primary:
       <<: *default
       url: <%= ENV['DATABASE_URL'] %>
     cache:
       <<: *cache
       url: <%= ENV['DATABASE_URL']&.+('_cache') %>
     queue:
       <<: *queue
       url: <%= ENV['DATABASE_URL']&.+('_queue') %>
     cable:
       <<: *cable
       url: <%= ENV['DATABASE_URL']&.+('_cable') %>
   

   That works fine although I feel like it is a bit cluttered for a new app. Maybe I could simplify this but I am unsure how. ~I guess I could dynamically set the database names like database: "myapp_#{Rails.env}_queue") and hoist those definitions in &queue etc. Edit: that doesn't actually work when creating or dropping multiple DBs at the same time~

3. Provide a command like bin/rails solid_queue:install:dev that sets up everything for your development environment (with a database.yml similar to point 2).

What do people think is the best way to play around with Solid Queue in development?

(I realize this question applies to Solid Cache and Solid Cable too since they also use a separate schema, so it may be more of a general Rails question. I can repost on the Rails forum or the Rails github discussions if needed)

[rosa]

Hey @jeromedalbert, good question! I think the best way to play around with Solid Queue (and Solid Cache) in development is to keep the databases separate as well.

I think your idea of an install command for dev or a commented out example is good, that'll help people for sure.

[jeromedalbert]

Related: https://github.com/rails/solid_queue/pull/334 and https://github.com/rails/solid_cable/pull/37 added instructions to use a single database. Although with those solutions you would have a single database for both development and production.

[jeromedalbert]

Just some updates on how I currently set up my database.yml if that can help anyone.

• For SQLite:

  default: &default
    adapter: sqlite3
    pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
    timeout: 5000
  
  databases: &databases
    primary:
      <<: *default
      database: storage/<%= Rails.env %>.sqlite3
    cache:
      <<: *default
      database: storage/<%= Rails.env %>_cache.sqlite3
      migrations_paths: db/cache_migrate
    queue:
      <<: *default
      database: storage/<%= Rails.env %>_queue.sqlite3
      migrations_paths: db/queue_migrate
    cable:
      <<: *default
      database: storage/<%= Rails.env %>_cable.sqlite3
      migrations_paths: db/cable_migrate
  
  development:
    <<: *databases
  
  test:
    <<: *default
    database: storage/test.sqlite3
  
  production:
    <<: *databases
  

• For database servers like Postgres or MySQL:

  (not extensively tested in production but it seems to work when trying it out in my local)

  default: &default
    adapter: postgresql
    encoding: unicode
    pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
  
  databases: &databases
    primary: &primary
      <<: *default
      database: myapp_<%= Rails.env %>
      password: <%= ENV["MYAPP_DATABASE_PASSWORD"] %>
    cache:
      <<: *primary
      database: myapp_<%= Rails.env %>_cache
      migrations_paths: db/cache_migrate
    queue:
      <<: *primary
      database: myapp_<%= Rails.env %>_queue
      migrations_paths: db/queue_migrate
    cable:
      <<: *primary
      database: myapp_<%= Rails.env %>_cable
      migrations_paths: db/cable_migrate
  
  development:
    <<: *databases
  
  test:
    <<: *default
    database: myapp3_test
  
  production:
    <<: *databases
  

• For database servers like Postgres or MySQL with a DATABASE_URL:

  (using dotenv in development with a .env that contains DATABASE_URL=postgres://localhost:5432/myapp_development for Postgres, or DATABASE_URL=mysql://root@localhost:3306/myapp_development for MySQL)

  default: &default
    adapter: postgresql
    encoding: unicode
    pool: <%= ENV.fetch('RAILS_MAX_THREADS') { 5 } %>
  
  databases: &databases
    primary:
      <<: *default
      url: <%= ENV['DATABASE_URL'] %>
    cache:
      <<: *default
      url: <%= URI.parse(ENV['DATABASE_URL']).tap { |u| u.path += '_cache' } if ENV['DATABASE_URL'] %>
      migrations_paths: db/cache_migrate
    queue:
      <<: *default
      url: <%= URI.parse(ENV['DATABASE_URL']).tap { |u| u.path += '_queue' } if ENV['DATABASE_URL'] %>
      migrations_paths: db/queue_migrate
    cable:
      <<: *default
      url: <%= URI.parse(ENV['DATABASE_URL']).tap { |u| u.path += '_cable' } if ENV['DATABASE_URL'] %>
      migrations_paths: db/cable_migrate
  
  development:
    <<: *databases
  
  test:
    <<: *default
    database: myapp3_test
  
  production:
    <<: *databases
  

Reusing the same config in development and prod makes for a shorter and simpler database.yml so I've been pretty happy with this.

[rosa]

I forgot to close this one when proper development instructions were added.