octopus
octopus copied to clipboard
thiagopradi
rake db:migrate should run only on master (shouldn't try to run on slaves)
I have fully replicated MySQL Master-Slave databases. I just want to run rake db:migrate command on master shard.
If Octopus try to run rake db:migrate command on slaves, MySQL Slaves stop working.
My migration files and models not include any using(:shard) blocks.
I just want to send all write queries and migration commands to master, and all reads queries to slaves. MySQL is handling replication job.
Detailed information are listed below.
Problem steps:
- run
rake db:migrate RAILS_ENV=development - Octopus try to run migration command. Trying order: (slave1, slave2, slave3, master) (I dont want to run migration command on slaves but Octopus trying...)
- This time MySQL slaves stop working. Because MySQL configured as slaves.
Example MySQL Error:
[ERROR] Slave SQL: Error 'Table 'schema_migrations' already exists' on query. Default database: 'ustad_development'. Query: 'CREATE TABLE `schema_migrations` (`version` varchar(255) NOT NULL) ENGINE=InnoDB', Error_code: 1050
[Warning] Slave: Table 'schema_migrations' already exists Error_code: 1050
[ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql
-bin.000001' position 310
As a result, I just want to run rake db:migrate command on master shard. What am i missing?
# config/shards.yml
octopus:
replicated: true
fully_replicated: true
environments:
- development
development:
slave1:
adapter: mysql2
encoding: utf8
pool: 10
username: <%= ENV['MYSQL_SLAVE_1_ENV_MYSQL_USER'] %>
password: <%= ENV['MYSQL_SLAVE_1_ENV_MYSQL_PASS'] %>
host: <%= ENV['MYSQL_SLAVE_1_PORT_3306_TCP_ADDR'] %>
port: <%= ENV['MYSQL_SLAVE_1_PORT_3306_TCP_PORT'] %>
database: foo_development
slave2:
adapter: mysql2
encoding: utf8
pool: 10
username: <%= ENV['MYSQL_SLAVE_2_ENV_MYSQL_USER'] %>
password: <%= ENV['MYSQL_SLAVE_2_ENV_MYSQL_PASS'] %>
host: <%= ENV['MYSQL_SLAVE_2_PORT_3306_TCP_ADDR'] %>
port: <%= ENV['MYSQL_SLAVE_2_PORT_3306_TCP_PORT'] %>
database: foo_development
slave3:
adapter: mysql2
encoding: utf8
pool: 10
username: <%= ENV['MYSQL_SLAVE_3_ENV_MYSQL_USER'] %>
password: <%= ENV['MYSQL_SLAVE_3_ENV_MYSQL_PASS'] %>
host: <%= ENV['MYSQL_SLAVE_3_PORT_3306_TCP_ADDR'] %>
port: <%= ENV['MYSQL_SLAVE_3_PORT_3306_TCP_PORT'] %>
database: foo_development
# config/database.yml
production:
adapter: mysql2
encoding: utf8
pool: 10
database: foo
username: <%= Rails.application.secrets.database['username'] %>
password: <%= Rails.application.secrets.database['password'] %>
host: <%= Rails.application.secrets.database['host'] %>
port: <%= Rails.application.secrets.database['port'] %>
development:
adapter: mysql2
encoding: utf8
pool: 10
database: foo_development
username: <%= Rails.application.secrets.database['username'] %>
password: <%= Rails.application.secrets.database['password'] %>
host: <%= Rails.application.secrets.database['host'] %>
port: <%= Rails.application.secrets.database['port'] %>
test:
adapter: mysql2
encoding: utf8
pool: 10
database: foo_test
username: <%= Rails.application.secrets.database['username'] %>
password: <%= Rails.application.secrets.database['password'] %>
host: <%= Rails.application.secrets.database['host'] %>
port: <%= Rails.application.secrets.database['port'] %>
@sobrinho: Can you help me? :(
Does the comment from velles here https://github.com/thiagopradi/octopus/issues/345 work for you?
@anhkind No, this method adds the master shard at the end of the shards list. So, Octopus is trying in that order: slave1, slave2, slave3, master.
Can you paste the content of the migration file that failed the migration here?
# migration file example
class CreateUsers < ActiveRecord::Migration
def change
create_table :users do |t|
t.string :first_name, null: false, limit: 40
t.string :last_name, null: false, limit: 40
t.string :email, null: false, limit: 100
t.string :password_digest, null: false
t.string :picture, limit: 50
t.string :cid, limit: 20
t.string :cell_phone, limit: 20
t.string :address, limit: 250
t.string :utype, null: false, limit: 20
t.integer :institution_id, null: false
t.integer :city_id, null: false
t.integer :gender, null: false, limit: 1, default: 0
t.integer :blood_group, null: false, limit: 1, default: 0
t.integer :status, null: false, limit: 1, default: 0
t.date :birth_date
t.timestamps null: false
end
add_index :users, :institution_id
add_index :users, :utype
add_index :users, :email, unique: true
end
end
All files like this. They only create the table.I didn't make any changes on migration files for the Octopus.
Are you running version 0.8.5 or from github's master branch? Can you try the master branch to see if it helps?
I think I'm having a related issue where I run rake db:migrate:status and it shows all up but the last, newly created migration. And then if I run rake db:migrate it errors because it's trying to run the very first migration.
I'm running postgres and Octopus 0.9.0. The error output from rake db:migrate is because of a now mis-named migration that was run years ago. In every other environment, it doesn't matter as it's already been run.
Here is the slimmed-down output:
root@402af45ba5c7:/paid# rake db:migrate:status --trace
** Invoke db:migrate:status (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke db:load_config (first_time)
** Execute db:load_config
** Execute db:migrate:status
database: app
Status Migration ID Migration Name
--------------------------------------------------
up 20141017224634 Create extensions
up 20141117224634 Create API keys
...
up 20170418194101 Add business support phone to accounts
down 20170426043725 Add temp to transactions
root@402af45ba5c7:/paid# rake db:migrate --trace
** Invoke db:migrate (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke db:load_config (first_time)
** Execute db:load_config
** Execute db:migrate
rake aborted!
NameError: uninitialized constant CreateAPIKeys
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activesupport-4.2.0/lib/active_support/inflector/methods.rb:261:in `const_get'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activesupport-4.2.0/lib/active_support/inflector/methods.rb:261:in `block in constantize'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activesupport-4.2.0/lib/active_support/inflector/methods.rb:259:in `each'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activesupport-4.2.0/lib/active_support/inflector/methods.rb:259:in `inject'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activesupport-4.2.0/lib/active_support/inflector/methods.rb:259:in `constantize'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activesupport-4.2.0/lib/active_support/core_ext/string/inflections.rb:66:in `constantize'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activerecord-4.2.0/lib/active_record/migration.rb:775:in `load_migration'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activerecord-4.2.0/lib/active_record/migration.rb:770:in `migration'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/migration.rb:146:in `shards'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/migration.rb:96:in `block in migrations_with_octopus'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/migration.rb:96:in `select'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/migration.rb:96:in `migrations_with_octopus'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activerecord-4.2.0/lib/active_record/migration.rb:1001:in `target'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activerecord-4.2.0/lib/active_record/migration.rb:948:in `migrate'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/migration.rb:86:in `migrate_with_octopus'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activerecord-4.2.0/lib/active_record/migration.rb:820:in `up'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/migration.rb:110:in `up_with_octopus'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activerecord-4.2.0/lib/active_record/migration.rb:798:in `migrate'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/migration.rb:104:in `block in migrate_with_octopus'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/proxy.rb:246:in `block (2 levels) in run_queries_on_shard'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/proxy.rb:509:in `using_shard'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/proxy.rb:245:in `block in run_queries_on_shard'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/proxy.rb:492:in `keeping_connection_proxy'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/proxy.rb:244:in `run_queries_on_shard'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/proxy.rb:253:in `block in send_queries_to_multiple_shards'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/proxy.rb:252:in `map'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/proxy.rb:252:in `send_queries_to_multiple_shards'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ar-octopus-0.8.6/lib/octopus/migration.rb:103:in `migrate_with_octopus'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activerecord-4.2.0/lib/active_record/tasks/database_tasks.rb:137:in `migrate'
/root/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/activerecord-4.2.0/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/task.rb:240:in `call'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/task.rb:240:in `block in execute'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/task.rb:235:in `each'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/task.rb:235:in `execute'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/task.rb:179:in `block in invoke_with_call_chain'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/monitor.rb:211:in `mon_synchronize'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/task.rb:172:in `invoke_with_call_chain'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/task.rb:165:in `invoke'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:150:in `invoke_task'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:106:in `block (2 levels) in top_level'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:106:in `each'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:106:in `block in top_level'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:115:in `run_with_threads'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:100:in `top_level'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:78:in `block in run'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:176:in `standard_exception_handling'
/root/.rbenv/versions/2.2.1/lib/ruby/2.2.0/rake/application.rb:75:in `run'
/root/.rbenv/versions/2.2.1/bin/rake:33:in `<main>'
Tasks: TOP => db:migrate
@ryanwjackson Nope... I still have to scroll my terminal up for 20 seconds to see what happened in my migration.
So, I actually figured out mine. It was an issue with how the migrations were being loaded by Octopus. We had added some inflections, which meant that those classes couldn't be loaded properly. Normal rails migrations apparently doesn't attempt to load in the same way, but Octopus does. So renaming the classes properly worked.
That said, are you doing replication or sharding?
see https://github.com/thiagopradi/octopus/issues/345#issuecomment-453046283
