Multiple database support

[bradens]

Hey there, wondering if this supports multiple databases in rails 6.

https://guides.rubyonrails.org/active_record_multiple_databases.html

I've got a primary and secondary db setup, and currently I it would appear that when running

rake db:migrate:with_data

rake aborted!
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "data_migrations" does not exist
-e:1:in `<main>'

Since the secondary database doesn't have the data_migrations table in it.

Rails 6 added the migrations_paths option to database.yml, so you can have the different migrations separated. Should data-migrate support something similar?

If so, i'd be happy to take a whack at a PR to support this.

[ilyakatz]

Hi @bradens, we haven't done anything special to support multiple databases. However, we do support different migrations_paths (see the docs ) . Does this solve your problem?

[happy-dariush]

Hi there! @ilyakatz I ran into the same issue as @bradens I found out that the problem with multiple databases is that at this point ActiveRecord::Base.connection refers to a secondary rather than a primary database. And there is no doubt that there is no data_migrations table in this database.

[phrsouza]

I believe that my issue is related somehow to the fact that data-migrate does not support multiple databases.

In my case, I'm using paper_trail gem with a custom version model that points to another database.

Whenever I try to apply a data migration over a model that uses paper_trail it fails saying that the relation versions do not exist.

PG::UndefinedTable: ERROR:  relation "versions" does not exist
LINE 8:  WHERE a.attrelid = '"versions"'::regclass

It would be great to have some support to multiple databases =)

[ilyakatz]

oh, that's an interesting find, @happy-dariush, so would it help if there was a config for which connection to use? something like

DataMigrate.configure do |config|
  config.db_connection = ActiveRecord::Base.connection
end

and gives you ability to update what connection is used?

[alvir]

@ilyakatz I assume we should have something like in Rails.

ActiveRecord::Base.configurations.configs_for(env_name: ActiveRecord::Tasks::DatabaseTasks.env).each do |db_config|
  ActiveRecord::Base.establish_connection(db_config.config)

I am willing to add it but have no enough time. And I guess it will be good if we create data_migrations table for each DB and data_schema.rb files for each DB. Maybe it will be good to have data_migrations_path in database.yml for each connection as it that place where we set config for different DBs.

[ilyakatz]

I'd rather not touch database.yml as this file belongs to a different gem. I'll take a look, but I'm tempted to go with my current config since it's already in place. Unless of course, you create a PR first, then I'm willing to reconsider ;)

[alvir]

I'd rather not touch database.yml as this file belongs to a different gem

I guess you're right. I would not like to mix configs and I guess database.yml is not so good to follow it. Probably a common hash looks simplier and clearer

[koriel-angelswing]

Same here. We are using separate database schemas up to the version to API like v1 and v2. So as far as I understood through this thread, we cannot use separated data_migrations table according to the database version like below right?

data_migrations in v1 schema for v1 database. data_migrations in v2 schema for v2 database.

[trammel]

Is there a solution for this? We've just met the same issue after adding a second database configuration, and adding migrations causes the active connection to switch to the second database, and then rake --trace db:migrate:with_data fails

Caused by:
PG::UndefinedTable: ERROR:  relation "data_migrations" does not exist
LINE 1: SELECT "data_migrations"."version" FROM "data_migrations" OR...
...
/home/jonpad/git/rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/data_migrate-6.3.0/lib/data_migrate/schema_dumper.rb:33:in `initialize'
/home/jonpad/git/rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/data_migrate-6.3.0/lib/data_migrate/schema_dumper.rb:12:in `new'
/home/jonpad/git/rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/data_migrate-6.3.0/lib/data_migrate/schema_dumper.rb:12:in `dump'
/home/jonpad/git/rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/data_migrate-6.3.0/tasks/databases.rake:349:in `block (3 levels) in <main>'
/home/jonpad/git/rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/data_migrate-6.3.0/tasks/databases.rake:348:in `open'
/home/jonpad/git/rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/data_migrate-6.3.0/tasks/databases.rake:348:in `block (2 levels) in <main>'
/home/jonpad/git/rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'

[ilyakatz]

I tried to add this functionality a while ago but ran into issues. I've been pretty busy lately and I'm no longer actively working with Ruby/rails. Is everyone on the thread interested in trying to give it a shot?

[drewbaumann]

I tried to add this functionality a while ago but ran into issues. I've been pretty busy lately and I'm no longer actively working with Ruby/rails. Is everyone on the thread interested in trying to give it a shot?

I am tempted, but haven't looked deeply into the problem/solution. Is there an agreed upon approach to solving this?

[ilyakatz]

I was thinking about this approach https://github.com/ilyakatz/data-migrate/issues/147#issuecomment-614902109 but open to other approaches.

[drewbaumann]

I tried to add this functionality a while ago but ran into issues. I've been pretty busy lately and I'm no longer actively working with Ruby/rails. Is everyone on the thread interested in trying to give it a shot?

I am tempted, but haven't looked deeply into the problem/solution. Is there an agreed upon approach to solving this?

I'm unfortunately not going to have time to get to this.

[chaunce]

I have a branch that appears to properly run migrations across all configured connections, in the same way that schema migrations work. I don't have a lot of time right now to test and fix specs, but if anyone else wants to contribute we could get a pull request submitted once this is cleaned up https://github.com/chaunce/data-migrate/tree/multiple_connection_support

[chaunce]

I have fixed the specs and created a pull request. Any testing and feedback anyone can offer would be helpful https://github.com/ilyakatz/data-migrate/pull/249

[ilyakatz]

thanks for the fix!