Adds Before Retry Proc

[wilkosz]

Agenda

Rails does not provide the functionality to update database credentials within ActiveRecord::Base.connection_pool (allow rotating credentials) during runtime.

This could be facilitated by rescuing Mysql2::Error::ConnectionError and manually updating ActiveRecord::Base.configurations[Rails.env] and then clearing/reset all connections in ActiveRecord::Base.connection_pool but this results in transaction downtime.

Solution

Add before retry proc allowing an update of credentials during Mysql2::Error::ConnectionError

Example Proc for updating rotating credentials

Proc.new do |e, sql, conn|
        return unless ['production_db', 'staging', 'production'].include?(Rails.env) &&
          e.is_a?(Mysql2::Error::ConnectionError) &&
          e.message =~ /access\sdenied/i
        # Get fresh credentials
        client = Aws::SecretsManager::Client.new(...)
        db = eval(client.get_secret_value(secret_id: 'prod/database/rotating/1').secret_string)
        db.each_pair do |k, v|
          ENV["DB_#{k.to_s.underscore.upcase}"] = v.to_s
        end
        # Update database connections and update existing connection
        config =
          ActiveRecord::Base.configurations[Rails.env] =
          Rails.application.config.database_configuration[Rails.env] =
          YAML::load(ERB.new(File.read(File.join(Rails.root, "config/database.yml"))).result)[Rails.env]

        # Ensure all new connections are using updated credentials
        cp             = ActiveRecord::Base.connection_pool
        cp_spec        = cp.instance_variable_get(:@spec)
        cp_spec_config = cp_spec.instance_variable_get(:@config)
        cp_spec_config.merge(config)
        cp_spec.instance_variable_set(:@config, cp_spec_config)

        # Ensure all existing connection are using updated
        ActiveRecord::Base.connection_pool.connections.each do |c|
          c_config = c.instance_variable_get(:@config)
          c_config.merge!(config)
          c.instance_variable_set(:@config, c_config)
          c.send :disconnect!
          c.send :connect
        end
      end

[wilkosz]

@winebarrel @sikachu @amatsuda @ssig33 @duffyjp Can someone please review?