Exception in tornado when calling 'ceph.maintenance.upgrade'

[jschmid1]

Running salt-run state.orch ceph.maintenance.upgrade errors out after ceph.updates.salt with a default error screen. From the logs I can see the following.

2018-11-06 13:54:30,997 [salt.transport.zeromq:393 ][DEBUG   ][58454] Setting zmq_reconnect_ivl_max to '11000ms'                                                                                                                                                                 
2018-11-06 13:54:30,998 [tornado.application:637 ][ERROR   ][58454] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f92d5893048>, <tornado.concurrent.Future object at 0x7f92e17b4e10>)                                                       
Traceback (most recent call last):                                                       
  File "/usr/lib64/python3.6/site-packages/tornado/ioloop.py", line 604, in _run_callback                                                                                                                                                                                        
    ret = callback()                                                                                                                                                                                                                                                             
  File "/usr/lib64/python3.6/site-packages/tornado/stack_context.py", line 276, in null_wrapper                                                                                                                                                                                  
    return fn(*args, **kwargs)                                                                                                                                                                                                                                                   
  File "/usr/lib64/python3.6/site-packages/tornado/ioloop.py", line 625, in _discard_future_result
    future.result()                                                                    
  File "/usr/lib64/python3.6/site-packages/tornado/concurrent.py", line 237, in result  
    raise_exc_info(self._exc_info)                                                                                          
  File "<string>", line 4, in raise_exc_info                                                                                                                                                                                                                                     
  File "/usr/lib64/python3.6/site-packages/tornado/gen.py", line 292, in wrapper                                                                                                                                                                                                 
    result = func(*args, **kwargs)                                                                                                                                                                                                                                               
TypeError: run() got an unexpected keyword argument 'async'                                                                                                                                                                                                                       
2018-11-06 13:54:31,000 [tornado.application:637 ][ERROR   ][58454] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f92d58930d0>, <tornado.concurrent.Future object at 0x7f92d58920f0>)                                                       
Traceback (most recent call last):                                                     
  File "/usr/lib64/python3.6/site-packages/tornado/ioloop.py", line 604, in _run_callback       
    ret = callback()                                                                                                                                                                                                                                                             
  File "/usr/lib64/python3.6/site-packages/tornado/stack_context.py", line 276, in null_wrapper                                                                                                                                                                                  
    return fn(*args, **kwargs)                                                                                                                                                                                                                                                   
  File "/usr/lib64/python3.6/site-packages/tornado/ioloop.py", line 625, in _discard_future_result                                                                                                                                                                                
    future.result()                                                          
  File "/usr/lib64/python3.6/site-packages/tornado/concurrent.py", line 237, in result 
    raise_exc_info(self._exc_info)                                                                                                                                                                                                                                               
  File "<string>", line 4, in raise_exc_info                                                                                                                                                                                                                                     
  File "/usr/lib64/python3.6/site-packages/tornado/gen.py", line 292, in wrapper                                                                                                                                                                                                 
    result = func(*args, **kwargs)                                                                                                                                                                                                                                                
TypeError: run() got an unexpected keyword argument 'async'  

Salt version report:

Salt Version:
           Salt: 2018.3.0

Dependency Versions:
           cffi: 1.11.2
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.4
   mysql-python: Not Installed
      pycparser: 2.17
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.5 (default, Apr 05 2018, 13:30:06) [GCC]
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 17.0.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.3

System Versions:
           dist:
         locale: UTF-8
        machine: x86_64
        release: 4.12.14-25.22-default
         system: Linux
        version: Not Installed

[jschmid1]

It seems that ceph.updates.salt

triggers

service.restart on salt-minion

which correctly triggers a restart of the salt-minion. Other than in the version before, this causes issues in the transport.

2018-11-07 16:00:31,181 [salt.loaded.int.module.cmdmod:385 ][INFO    ][28591] Executing command ['systemctl', 'status', 'salt-minion.service', '-n', '0'] in directory '/root'
2018-11-07 16:00:31,203 [salt.loaded.int.module.cmdmod:1827][DEBUG   ][28591] stdout: * salt-minion.service - The Salt Minion
   Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-11-07 15:55:19 UTC; 5min ago                                                                                                                                                                                                                                                                                                     
 Main PID: 28180 (salt-minion)
    Tasks: 10 (limit: 4915)
   CGroup: /system.slice/salt-minion.service
           |-28180 /usr/bin/python3 /usr/bin/salt-minion                                                                                                                                                                                                                                                                                                                    
           |-28185 /usr/bin/python3 /usr/bin/salt-minion
           |-28188 /usr/bin/python3 /usr/bin/salt-minion                                                                                                                                                                                                                                                                                                                    
           |-28591 /usr/bin/python3 /usr/bin/salt-minion
           `-29140 systemctl status salt-minion.service -n 0                                                                                                                                                                                                                                                                                                                
2018-11-07 16:00:31,215 [salt.loaded.int.module.cmdmod:385 ][INFO    ][28591] Executing command ['systemd-run', '--scope', 'systemctl', 'restart', 'salt-minion.service'] in directory '/root'                                                                                                                                                                              
2018-11-07 16:00:31,294 [salt.utils.parsers:1045][WARNING ][28185] Minion received a SIGTERM. Exiting.

and causes the orchestration to fail.

[smithfarm]

@jschmid1 I guess this is why the new tier2/upgrade doesn't pass for me.

[jschmid1]

@jschmid1 I guess this is why the new tier2/upgrade doesn't pass for me.

Right, this is where this issue arose from