slowpoke timeout causing deployment build timeout and crash

[donrestarone]

trafficstars

Describe the bug during application release in Capistrano, there is a chance of timeout and release failure

affecting Nikean.org / a-toi.ca

2023-03-22T12:09:41.6106806Z 03:12 deploy:assets:rsync
2023-03-22T12:09:41.6108253Z       01 rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/
2023-03-22T12:11:51.5737152Z       01 ssh: connect to host *** port 22: Connection timed out
2023-03-22T12:11:51.5746580Z       01 rsync: connection unexpectedly closed (0 bytes received so far) [sender]
2023-03-22T12:11:51.5749698Z       01 rsync error: unexplained error (code 255) at io.c(231) [sender=3.2.7]
2023-03-22T12:11:51.5752143Z #<Thread:0x0000563d65358588@/home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/runners/parallel.rb:10 run> terminated with exception (report_on_exception is true):
2023-03-22T12:11:51.5755691Z /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute': Exception while executing as ubuntu@***: rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/ exit status: 65280 (SSHKit::Runner::ExecuteError)
2023-03-22T12:11:51.5758077Z rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/ stdout: Nothing written
2023-03-22T12:11:51.5760161Z rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/ stderr: ssh: connect to host *** port 22: Connection timed out
2023-03-22T12:11:51.5761656Z rsync: connection unexpectedly closed (0 bytes received so far) [sender]
2023-03-22T12:11:51.5762857Z rsync error: unexplained error (code 255) at io.c(231) [sender=3.2.7]
2023-03-22T12:11:51.5764495Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'
2023-03-22T12:11:51.5767110Z /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/command.rb:97:in `exit_status=': rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/ exit status: 65280 (SSHKit::Command::Failed)
2023-03-22T12:11:51.5769493Z rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/ stdout: Nothing written
2023-03-22T12:11:51.5771550Z rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/ stderr: ssh: connect to host *** port 22: Connection timed out
2023-03-22T12:11:51.5773025Z rsync: connection unexpectedly closed (0 bytes received so far) [sender]
2023-03-22T12:11:51.5774215Z rsync error: unexplained error (code 255) at io.c(231) [sender=3.2.7]
2023-03-22T12:11:51.5775825Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/local.rb:59:in `block in execute_command'
2023-03-22T12:11:51.5778128Z 	from /opt/hostedtoolcache/Ruby/2.6.6/x64/lib/ruby/2.6.0/open3.rb:219:in `popen_run'
2023-03-22T12:11:51.5779383Z 	from /opt/hostedtoolcache/Ruby/2.6.6/x64/lib/ruby/2.6.0/open3.rb:101:in `popen3'
2023-03-22T12:11:51.5781015Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/local.rb:44:in `execute_command'
2023-03-22T12:11:51.5782871Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:148:in `block in create_command_and_execute'
2023-03-22T12:11:51.5785139Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:148:in `tap'
2023-03-22T12:11:51.5786917Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:148:in `create_command_and_execute'
2023-03-22T12:11:51.5788795Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:80:in `execute'
2023-03-22T12:11:51.5790720Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/capistrano-local-precompile-1.2.0/lib/capistrano/local_precompile.rb:49:in `block (6 levels) in <top (required)>'
2023-03-22T12:11:51.5792666Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/capistrano-local-precompile-1.2.0/lib/capistrano/local_precompile.rb:45:in `each'
2023-03-22T12:11:51.5794597Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/capistrano-local-precompile-1.2.0/lib/capistrano/local_precompile.rb:45:in `block (5 levels) in <top (required)>'
2023-03-22T12:11:51.5796478Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:31:in `instance_exec'
2023-03-22T12:11:51.5798163Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:31:in `run'
2023-03-22T12:11:51.5799833Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/capistrano-3.16.0/lib/capistrano/dsl.rb:76:in `run_locally'
2023-03-22T12:11:51.5801759Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/capistrano-local-precompile-1.2.0/lib/capistrano/local_precompile.rb:38:in `block (4 levels) in <top (required)>'
2023-03-22T12:11:51.5803617Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:31:in `instance_exec'
2023-03-22T12:11:51.5805292Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:31:in `run'
2023-03-22T12:11:51.5807057Z 	from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
2023-03-22T12:11:51.5808703Z (Backtrace restricted to imported tasks)
2023-03-22T12:11:51.5809657Z cap aborted!
2023-03-22T12:11:51.5822575Z SSHKit::Runner::ExecuteError: Exception while executing as ubuntu@***: rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/ exit status: 65280
2023-03-22T12:11:51.5824719Z rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/ stdout: Nothing written
2023-03-22T12:11:51.5826794Z rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/ stderr: ssh: connect to host *** port 22: Connection timed out
2023-03-22T12:11:51.5840090Z rsync: connection unexpectedly closed (0 bytes received so far) [sender]
2023-03-22T12:11:51.5847996Z rsync error: unexplained error (code 255) at io.c(231) [sender=3.2.7]
2023-03-22T12:11:51.5849065Z 
2023-03-22T12:11:51.5849457Z 
2023-03-22T12:11:51.5849966Z Caused by:
2023-03-22T12:11:51.5852032Z SSHKit::Command::Failed: rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/ exit status: 65280
2023-03-22T12:11:51.5854162Z rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/ stdout: Nothing written
2023-03-22T12:11:51.5856619Z rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/ stderr: ssh: connect to host *** port 22: Connection timed out
2023-03-22T12:11:51.5858693Z rsync: connection unexpectedly closed (0 bytes received so far) [sender]
2023-03-22T12:11:51.5860919Z rsync error: unexplained error (code 255) at io.c(231) [sender=3.2.7]
2023-03-22T12:11:51.5861642Z 
2023-03-22T12:11:51.5862153Z Tasks: TOP => deploy:assets:rsync
2023-03-22T12:11:51.5863231Z (See full trace by running task with --trace)
2023-03-22T12:11:51.5865439Z The deploy has failed with an error: Exception while executing as ubuntu@***: rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/ exit status: 65280
2023-03-22T12:11:51.5867560Z rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/ stdout: Nothing written
2023-03-22T12:11:51.5869708Z rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/ stderr: ssh: connect to host *** port 22: Connection timed out
2023-03-22T12:11:51.5871286Z rsync: connection unexpectedly closed (0 bytes received so far) [sender]
2023-03-22T12:11:51.5872562Z rsync error: unexplained error (code 255) at io.c(231) [sender=3.2.7]
2023-03-22T12:11:51.5873264Z 
2023-03-22T12:11:51.5873339Z 
2023-03-22T12:11:51.5873763Z ** DEPLOY FAILED
2023-03-22T12:11:51.5874825Z ** Refer to log/capistrano.log for details. Here are the last 20 lines:
2023-03-22T12:11:51.5875547Z 
2023-03-22T12:11:51.5875621Z 
2023-03-22T12:11:51.5876114Z  DEBUG [ed9a633b] 	  application (1.44 MiB)
2023-03-22T12:11:51.5876685Z 
2023-03-22T12:11:51.5877284Z  DEBUG [ed9a633b] 	      css/application-5bbc6a2f.css
2023-03-22T12:11:51.5877907Z 
2023-03-22T12:11:51.5878523Z  DEBUG [ed9a633b] 	      js/application-0d74e5e1e127c6c8c59f.js
2023-03-22T12:11:51.5880487Z 
2023-03-22T12:11:51.5880932Z  DEBUG [ed9a633b] 	  comfy/admin/cms (758 KiB)
2023-03-22T12:11:51.5881455Z 
2023-03-22T12:11:51.5882046Z  DEBUG [ed9a633b] 	      js/comfy/admin/cms-69873a4cab68c11a8f77.js
2023-03-22T12:11:51.5882662Z 
2023-03-22T12:11:51.5883063Z  DEBUG [ed9a633b] 	  tribute (349 KiB)
2023-03-22T12:11:51.5883556Z 
2023-03-22T12:11:51.5884087Z  DEBUG [ed9a633b] 	      js/tribute-7b2b18e777aa09547aa7.js
2023-03-22T12:11:51.5884657Z 
2023-03-22T12:11:51.5885054Z  DEBUG [ed9a633b] 	  trix (247 KiB)
2023-03-22T12:11:51.5885500Z 
2023-03-22T12:11:51.5886014Z  DEBUG [ed9a633b] 	      js/trix-9f83f6e6b369171a8f10.js
2023-03-22T12:11:51.5886579Z 
2023-03-22T12:11:51.5886934Z  DEBUG [ed9a633b] 	
2023-03-22T12:11:51.5887341Z 
2023-03-22T12:11:51.5888709Z  DEBUG [ed9a633b] 	Child mini-css-extract-plugin node_modules/css-loader/dist/cjs.js??ref--5-1!node_modules/postcss-loader/src/index.js??ref--5-2!node_modules/select2/dist/css/select2.css:
2023-03-22T12:11:51.5889720Z 
2023-03-22T12:11:51.5890263Z  DEBUG [ed9a633b] 	    Entrypoint mini-css-extract-plugin = *
2023-03-22T12:11:51.5890871Z 
2023-03-22T12:11:51.5891265Z  DEBUG [ed9a633b] 	       2 modules
2023-03-22T12:11:51.5891719Z 
2023-03-22T12:11:51.5892065Z  DEBUG [ed9a633b] 	
2023-03-22T12:11:51.5892467Z 
2023-03-22T12:11:51.5893028Z   INFO [ed9a633b] Finished in 174.896 seconds with exit status 0 (successful).
2023-03-22T12:11:51.5893677Z 
2023-03-22T12:11:51.5894798Z   INFO [54f8f1f7] Running rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/ as runner@localhost
2023-03-22T12:11:51.5895702Z 
2023-03-22T12:11:51.5896720Z  DEBUG [54f8f1f7] Command: rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20230322120630/public/assets/
2023-03-22T12:11:51.5897554Z 
2023-03-22T12:11:51.5898142Z  DEBUG [54f8f1f7] 	ssh: connect to host *** port 22: Connection timed out
2023-03-22T12:11:51.5898787Z 
2023-03-22T12:11:51.5899431Z  DEBUG [54f8f1f7] 	rsync: connection unexpectedly closed (0 bytes received so far) [sender]
2023-03-22T12:11:51.5900135Z 
2023-03-22T12:11:51.5900757Z  DEBUG [54f8f1f7] 	rsync error: unexplained error (code 255) at io.c(231) [sender=3.2.7]
2023-03-22T12:11:51.5901452Z 
2023-03-22T12:11:51.5930214Z ##[error]Error: The process '/opt/hostedtoolcache/Ruby/2.6.6/x64/bin/bundle' failed with exit code 1
2023-03-22T12:11:51.6005863Z Post job cleanup.
2023-03-22T12:11:51.6557212Z SSH key in /home/runner/.ssh has been removed successfully.
2023-03-22T12:11:51.6649722Z Post job cleanup.
2023-03-22T12:11:52.0958892Z The IP 172.177.246.182 is removed
2023-03-22T12:11:52.1123628Z Post job cleanup.
2023-03-22T12:11:52.2690974Z [command]/usr/bin/git version
2023-03-22T12:11:52.2737310Z git version 2.39.2
2023-03-22T12:11:52.2783640Z Temporarily overriding HOME='/home/runner/work/_temp/64b68efd-9e44-480b-81b5-2fa30a52ea0d' before making global git config changes
2023-03-22T12:11:52.2785367Z Adding repository directory to the temporary git global config as a safe directory
2023-03-22T12:11:52.2790202Z [command]/usr/bin/git config --global --add safe.directory /home/runner/work/violet_rails/violet_rails
2023-03-22T12:11:52.2841672Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2023-03-22T12:11:52.2875906Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2023-03-22T12:11:52.3136117Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2023-03-22T12:11:52.3168829Z http.https://github.com/.extraheader
2023-03-22T12:11:52.3181530Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2023-03-22T12:11:52.3215566Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2023-03-22T12:11:52.3646515Z Cleaning up orphan processes