cbrain icon indicating copy to clipboard operation
cbrain copied to clipboard

Published 20 hours ago • verified publisher icon aces

Prefetch tasks git file revision info resolves #1248

Open MontrealSergiy opened this issue 2 years ago • 6 comments

Prefetches tasks git file revision info #1248

Note, this will fail e.g. if there are ghost tools (configs) from deleted boutiques.

I can add some rescues if that's an issue, or just clean up the db from outdated configs

MontrealSergiy avatar Aug 22 '22 15:08 MontrealSergiy

@MontrealSergiy works on my side reduce the execution of git command.

But you said:

Note, this will fail e.g. if there are ghost tools (configs) from deleted boutiques.

How it will failed, what's happen in this specific case.

natacha-beck avatar Aug 24 '22 10:08 natacha-beck

In a inconsistent db the feature can put down cbrain workers with exception NameError while constantizing. But purging db and reinstall of plugins helps. The exception, as the cause of worker death only reported in logs, w.r.t GUI a worker dies silently

BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/inflector/methods.rb:270:in `const_get'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/inflector/methods.rb:270:in `block in constantize'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/inflector/methods.rb:266:in `each'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/inflector/methods.rb:266:in `inject'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/inflector/methods.rb:266:in `constantize'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/core_ext/string/inflections.rb:66:in `constantize'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/tool.rb:74:in `cbrain_task_class'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/tool_config.rb:324:in `cbrain_task_class'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activerecord-5.0.7/lib/active_record/relation/delegation.rb:38:in `map'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activerecord-5.0.7/lib/active_record/relation/delegation.rb:38:in `map'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/bourreau_worker.rb:60:in `setup'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/worker.rb:384:in `main_loop'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/worker.rb:291:in `block in start'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/config/initializers/cbrain.rb:142:in `block (2 levels) in spawn_with_active_records'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/config/initializers/cbrain.rb:102:in `fork'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/config/initializers/cbrain.rb:102:in `block in spawn_with_active_records'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/config/initializers/cbrain.rb:96:in `fork'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/config/initializers/cbrain.rb:96:in `spawn_with_active_records'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/worker.rb:204:in `start'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/worker_pool.rb:76:in `create_or_find_pool'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/bourreau.rb:410:in `process_command_start_workers'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/remote_resource.rb:786:in `process_command'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/controllers/controls_controller.rb:123:in `process_command'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/controllers/controls_controller.rb:92:in `create'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/abstract_controller/base.rb:188:in `process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_controller/metal/rendering.rb:30:in `process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/callbacks.rb:97:in `__run_callbacks__'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/callbacks.rb:750:in `_run_process_action_callbacks'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/callbacks.rb:90:in `run_callbacks'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/abstract_controller/callbacks.rb:19:in `process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_controller/metal/rescue.rb:20:in `process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/notifications.rb:164:in `block in instrument'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/notifications.rb:164:in `instrument'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_controller/metal/params_wrapper.rb:248:in `process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activerecord-5.0.7/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/abstract_controller/base.rb:126:in `process'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionview-5.0.7/lib/action_view/rendering.rb:30:in `process'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_controller/metal.rb:190:in `dispatch'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_controller/metal.rb:262:in `dispatch'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/routing/route_set.rb:32:in `serve'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/journey/router.rb:39:in `block in serve'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/journey/router.rb:26:in `each'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/journey/router.rb:26:in `serve'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/routing/route_set.rb:727:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/rack-2.0.5/lib/rack/conditional_get.rb:38:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/cookies.rb:613:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activerecord-5.0.7/lib/active_record/migration.rb:553:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/callbacks.rb:38:in `block in call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/callbacks.rb:97:in `__run_callbacks__'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/callbacks.rb:750:in `_run_call_callbacks'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/callbacks.rb:90:in `run_callbacks'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/callbacks.rb:36:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/executor.rb:12:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/debug_exceptions.rb:49:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/railties-5.0.7/lib/rails/rack/logger.rb:36:in `call_app'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/railties-5.0.7/lib/rails/rack/logger.rb:24:in `block in call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/tagged_logging.rb:69:in `block in tagged'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/tagged_logging.rb:26:in `tagged'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/tagged_logging.rb:69:in `tagged'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/railties-5.0.7/lib/rails/rack/logger.rb:24:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/request_id.rb:24:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/executor.rb:12:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/static.rb:136:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/railties-5.0.7/lib/rails/engine.rb:522:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/puma-3.11.4/lib/puma/configuration.rb:225:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/puma-3.11.4/lib/puma/server.rb:632:in `handle_request'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/puma-3.11.4/lib/puma/server.rb:446:in `process_client'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/puma-3.11.4/lib/puma/server.rb:306:in `block in run'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/puma-3.11.4/lib/puma/thread_pool.rb:120:in `block in spawn_thread'
BourreauWorker-20855: Worker shutting down.```

MontrealSergiy avatar Aug 24 '22 14:08 MontrealSergiy

a rescue can prevent that, smth like (ToolConfig.where(:bourreau_id => @rr_id).map {|tc| tc.cbrain_task_class rescue nil} ).uniq.each { |klass| klass&.revision_info&.self_update },

The problem is more likely occur in a dev environment, when tool/config get renamed etc. But if exists on production an admin can delete broken tools/config/tasks manually

MontrealSergiy avatar Aug 24 '22 18:08 MontrealSergiy

According to your commet:

In a inconsistent db the feature can put down cbrain workers with exception NameError > while constantizing. But purging db and reinstall of plugins helps. The exception, as the cause of worker death only reported in logs, w.r.t GUI a worker dies silently

You should add rescue we cannot let worker die without any message. We cannot let worker dies just for that.

natacha-beck avatar Aug 25 '22 08:08 natacha-beck

What about the following:

  ToolConfig.where(:bourreau_id => @rr_id)
            .map {|tc| tc.cbrain_task_class rescue nil}
            .uniq
            .compact   # to remove nil
            .each { |klass| klass.revision_info.self_update }

natacha-beck avatar Aug 25 '22 10:08 natacha-beck

A bit longer than what I suggested above yet should work too

MontrealSergiy avatar Aug 25 '22 14:08 MontrealSergiy

@prioux your comments addressed

MontrealSergiy avatar Nov 02 '22 14:11 MontrealSergiy

if you like I can rebase

MontrealSergiy avatar Nov 02 '22 14:11 MontrealSergiy

OK I'll check this soon

prioux avatar Jan 25 '23 21:01 prioux

This seems ok to me except for one minor thing: you created your method as a110 and in the file you put that method between a090 and a100. Just put it at the proper place please.

prioux avatar Feb 02 '23 15:02 prioux

This seems ok to me except for one minor thing: you created your method as a110 and in the file you put that method between a090 and a100. Just put it at the proper place please.

ok, the method is moved toward the end

MontrealSergiy avatar Feb 02 '23 15:02 MontrealSergiy